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FOREWORD 


These proceedings contain the papers of the lADIS International Conference on 
Cognition and Exploratory Learning in the Digital Age (CELDA 2013), 22-24 October 
2013, which has been organized by the International Association for Development of 
the Information Society (lADIS), co-organized by The University of North Texas (UNT), 
sponsored by the Association for Educational Communication and Technologies 
(AECT), and endorsed by the Japanese Society for Information and Systems in 
Education (JSISE). 

The CELDA 2013 conference aims to address the main issues concerned with evolving 
learning processes and supporting pedagogies and applications in the digital age. There 
have been advances in both cognitive psychology and computing that have affected the 
educational arena. The convergence of these two disciplines is increasing at a fast pace 
and affecting academia and professional practice in many ways. 

Paradigms such as just-in-time learning, constructivism, student-centered learning and 
collaborative approaches have emerged and are being supported by technological 
advancements such as simulations, virtual reality and multi-agents systems. These 
developments have created both opportunities and areas of serious concerns. This 
conference aims to cover both technological as well as pedagogical issues related to 
these developments. Main tracks have been identified. However innovative 
contributions that do not easily fit into these areas will also be considered as long as 
they are directly related to the overall theme of the conference - cognition and 
exploratory learning in the digital age. 


The following areas are represented in the submissions for CELDA 2013: 


• Acquisition of expertise 

• Assessing progress of learning in 
complex domains 

• Assessment of exploratory learning 
approaches 

• Assessment of exploratory 
technologies 

• Cognition in education 

• Collaborative learning 

• Educational psychology 

• Exploratory technologies 
(simulations, VR, i-TV, etc.) 

• Just-in-time and Eearning-on- 
Demand 


• Eeamer communities and peer- 
support 

• Eeaming communities & Web service 
technologies Pedagogical issues 
related with learning objects 

• Eeaming paradigms in academia 

• Eeaming paradigms in the corporate 
sector 

• Eife-long learning 

• Student-centered learning 

• Technology and mental models 

• Technology 

• Eeaming and expertise 

• Virtual university 


The CELDA 2013 Conference received 102 submissions from more than 16 countries. 
Each submission was reviewed in a double -blind review process by at least two 
independent reviewers to ensure quality and maintain high standards. Out of the papers 
submitted, 39 were accepted as full papers for an acceptance rate of 38%; 23 were 
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accepted as short papers and 6 were accepted as reflection papers. Authors of the best 
published papers in the CELDA 2013 proceedings will be invited to publish extended 
versions of their papers in a special issue of the Knowledge Management & E-Leaming: 
An International Journal (KM&EE), http://www.kmel-joumal.org/ojs/index.php/online- 
publication. 

In addition to the presentation of full papers, short papers and reflection papers, the 
conference also includes a keynote presentation from an internationally distinguished 
researcher. We would therefore like to express our gratitude to Professor Kinshuk, 
Associate Dean of Eaculty of Science and Technology, NSERC/iCORE/Xerox/Markin 
Industrial Research Chair - School of Computing and Information Systems, Athabasca 
University, Canada, as the CEEDA 2013 keynote speaker. 

The conference will also include a panel entitled "Interactive Technologies for Teacher 
Training: Two Technology Approaches and Their Implications" with Julia Meritt, 
David Gibson, Rhonda Christensen, Gerald Knezek and Wilhelmina Savenye. 

A successful conference requires the effort of many individuals. We would like to thank 
the members of the Program Committee for their hard work in reviewing and selecting 
the papers that appear in this book. We are especially grateful to the authors who 
submitted their papers to this conference and to the presenters who provided the 
substance of the meeting. We wish to thank all members of our organizing committee. 

East but not least, we hope that participants enjoy Eort Worth, Texas and their time with 
colleagues from all over the world; we invite you all to next year of the International 
Conference on Cognition and Exploratory Eeaming in the Digital Age, CEEDA 2014 
that will be held in Oporto, Portugal. 

Pedro Isaias, Universidade Aberta (Portuguese Open University), Portugal 
Conference Chair 

Demetrios G. Sampson, University of Piraeus and CERTH, Greece 
J. Michael Spector, University of North Texas, USA 
Dirk Ifenthaler, Open Universities, Australia 
Program Co-Chairs 


Eort Worth, Texas, USA 
22 October 2013 
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KEYNOTE LECTURE 


UBIQUITOUS LEARNING ANALYTICS FOR ADAPTIVE AND 
AUTHENTIC INSTRUCTION 


Professor Kinshuk, 

Associate Dean of Faculty of Science and Technology, 
NSERC/iCORE/Xerox/Markin Industrial Research Chair - School of Computing 
and Information Systems, Athabasca University, Canada 


Abstract 

There is greater awareness in educational system regarding benefits authentic learning 
experiences bring to the learning process. As a result, ubiquitous educational 
environments have started to gain acceptance in mainstream education. These 
environments break the boundaries of the classroom and enable learning to take place in 
the contexts where learners are able to relate with the learning scenarios in their own 
living and work environments. While these environments enable extremely powerful 
learning experiences for the learners, they demand highly complex teacher intervention 
techniques for providing appropriate support to the learners in real-time. This talk will 
focus on an emerging area of "ubiquitous learning analytics" that gleans real-time 
information from learners' immediate context, caters for any ambiguities or temporary 
absence of certain information, makes sense of information by analyzing patterns of 
real-time vs. historical learner profile, and enables teachers to intervene in both 
individual and group learning processes that are taking place in authentic environments. 
With particular focus on adaptive learning, the talk will explore the diminishing 
boundaries of formal and informal learning, and the potential of location-dependent 
context-sensitive approaches that are emerging as successor of Web 2.0 paradigm. 
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PANEL 


INTERACTIVE TECHNOLOGIES EOR TEACHER TRAINING: 
TWO TECHNOLOGY APPROACHES AND THEIR 
IMPLICATIONS 


By Julia Meritt, David Gibson, Rhonda Christensen, 
Gerald Knezek and Wilhelmina Savenye 


Abstract 

A widespread problem in schools concerns classroom management. While pre- service 
teachers are generally well-prepared with regard to pedagogy and subject matter, they 
are often less well-prepared in the area of classroom management considered broadly to 
include managing students with different backgrounds and levels of understanding as 
well as students becoming disengaged and disruptive. Most teacher preparation 
programs include an internship requirement, which involves working under the 
supervision of an experienced teacher for a number of hours in a particular classroom 
that may or may not be representative of the classroom in which the graduating teacher 
will be placed. As a result, teacher induction programs are being developed to help 
integrate incoming teachers in the particular culture and context of their schools and 
classrooms. In spite of efforts to support teacher induction and better prepare teachers in 
the area of classroom management, the high attrition rate of teachers in their first five 
years suggests persistent problems. Many of these problems concern the challenges of 
classroom management. This session demonstrates two highly interactive technologies 
aimed at giving pre- service and new teachers improved training in the area of classroom 
management. Both technologies involve simulations since it is not practical to provide 
teachers with actual interactive experiences with the many different kinds of student 
behaviors they will encounter. One approach involves a Second Life environment in 
which pre- and in-service teachers play the role of a classroom teacher or students in a 
classroom. The students can be programmed to exhibit different behaviors and the 
teacher has to react to these unanticipated behaviors. After the SecondLife session, the 
class can de-brief the session and discuss the behaviors and how the teacher responded, 
including suggestions for alternative reactions on the part of the teacher. A similar 
environment is supported in simSchool. The simSchool environment is simpler to create 
for a particular session and can be used to support classroom management as well as 
professional development, activity design and development, and promote an 
understanding of a variety of student behaviors. After each environment is presented, 
researchers and a discussant will provide a constructive critique and comment on the 
challenges of assessment and other aspects of these highly innovative technologies for 
teachers. The session will close with questions, comments and suggestions for 
subsequent R&D from participants. 
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lADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2013) 


WORKING MEMORY INTERVENTION: A READING 
COMPREHENSION APPROACH 


Tracy L. Perry and Evguenia Malaia 

The University of Texas at Arlington, USA 


ABSTRACT 

Eor any complex mental task, people rely on working memory. Working memory capacity (WMC) is one predictor of 
success in learning. Historically, attempts to improve verbal WM through training have not been effective. This study 
provided elementary students with WM consolidation efficiency training to answer the question. Can reading 
comprehension be improved by strategic updating of WMC and utilization of episodic memory during reading? 

We report preliminary data from 10 5* grade students who took three pretests to measure each student’s 1) ability to 
comprehend sentences of varying lengths, 2) ability to decode English-like words (Snowling, 1986), 3) working memory 
capacity (Weschler, D, 2008). 5 students were chosen to receive a working memory training that consists of 2 sessions a 
week for 4 weeks. In the training sessions, the students were presented with a series of progressively longer sentences and 
were asked to choose a picture to demonstrate comprehension. All 10 students took 2 post-tests, alternate versions of the 
working memory capacity and sentence comprehension pre-tests. 

The working memory and sentence comprehension data were analyzed using a one-tailed t-test. A priori power analysis 
indicated that I need 35 subjects in each of the 2 groups to have 95% power for detecting a large sized effect when 
employing the traditional .05 criterion of statistical significance. 

I expect that this training will change the strategies elementary readers use to update their WMC while reading to 
understand progressively more complex sentences. If successful, this training will dramatically affect reader’s interest 
and confidence, and can serve as a model for elementary reading programs. 


KEYWORDS 

Working memory, reading comprehension, episodic memory 


1. INTRODUCTION 

Although multiple models of working memory have been explored since the mid -twentieth century, Baddeley 
and Hitch’s model is often referenced in current research addressing the relationship between working 
memory and reading comprehension (Gathercole, 1998). Baddeley (2003) states that, regarding his 
multi-component model compared to others, “in general, deviations from other models represent differences 
of emphasis and scope, rather than direct conflict.” A basic understanding of Baddeley’s working memory 
model is helpful in discussing the relationship between working memory and reading comprehension. 

Baddeley (2003) and Hitch’s working memory model originally consisted of three components, the 
central executive, the phonological loop, and the visuo-spatial sketchpad. The phonological loop stores 
auditory memory traces for a few seconds before they fade, unless an articulatory rehearsal strategy is 
applied to maintain the memory, while the visuo-spatial sketchpad acquires visual input, which is limited in 
capacity to about three to four objects (Baddeley, 2003). The phonological loop and the visuo-spatial 
sketchpad are regulated by the central executive, which Baddeley (2003) described as “the most important 
but least understood component of working memory.” Baddeley (2003) identified one function of the central 
executive component as resource control; the ability to allocate attentional resources when two or more 
mental activities are being performed simultaneously. As a result of his research into the central executive 
component of working memory, Baddeley (2001) added a fourth component, which he calls the episodic 
buffer. The episodic buffer is a “temporary multidimensional store... that allows a range of different 
subsystems to interact” and whose main function is to “bind together different sources of information to form 
integrated chunks.” (Baddeley, 2001) 
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Because the comprehension of language, whether spoken or written, “entails processing a sequence of 
symbols that is produced and perceived over time,” the relationship between comprehension and working 
memory becomes evident (Just and Carpenter, 1992). Reading in particular requires storage of “intermediate 
and final computations” as ideas are constructed and integrated from the “stream of successive words in a 
texf’ (Just and Carpenter, 1992). 

Swanson (1992) studied working memory in skilled and less-skilled readers to determine if differences 
are related to a general system or not. Of interest to Swanson’s study are two theories regarding the 
relationship between working memory and poor reading skills. First, in Daneman’s (1980) model, working 
memory resources are related to reading skill, in that readers with inefficient skills overload working memory 
capacity, while efficient readers have adequate working memory resources left to store reading products. This 
would mean that “individual differences in working memory performances are tied to the specific processing 
task of reading,” not differences in working memory capacity (Daneman, 1980). 

A second model of interest to Swanson (1992) was Turner and Engle’s, which suggests that the working 
memory capacity of poor readers is smaller than that of strong readers. This model hypothesizes that working 
memory capacity is independent of reading skill, and therefore weaker working capacity is not a consequence 
of poor reading skills, but that the lower capacity leaves fewer resources available for performing reading and 
non-reading tasks (Swanson, 1992). 

Swanson (1992) reasoned that the inconsistency might be attributed to two issues: 1) “the psychometric 
qualities of the working memory tasks” which suggests “that individual differences in the relationship 
between working memory and reading reflect a number of factors other than working memory operations” 
and 2) “previous studies have relied... on single measures to reflect verbal or nonverbal working memory 
operations.” Swanson’s exploration of these issues “does not eliminate the possibility that processing 
efficiency at a language level drives this relationship between working memory and cognitive performance” 
(Swanson, 1992). While Swanson’s work did not conclusively support either Daneman or Turner and Engle, 
“the findings... suggest that working memory performance can be enhanced and that such performance 
appears to improve predictions of reading” (Swanson, 1992). 

Montgomery, Magimairaj and O’Malley (2008) also report a correlation between working memory and 
children’s ability to understand complex sentences.. Their study suggests that “comprehension accuracy for 
complex sentences was significantly associated with two of the three working memory mechanisms - 
attentional resource control/allocation and speed of processing” (Montgomery, Magimaiaj and O’Malley, 
2008). These results support Turner and Engle’s model which highlights the role of working memory 
capacity in reading comprehension, inviting further investigation into working memory interventions. 

Daneman and Carpenter (1983) also studied how the integration of information between and within 
sentences is related to working memory. Their research explored how integration processes “may stress the 
limits of working memory capacity and contribute to the purging of verbatim wording” on which readers rely 
when comprehending over sentence boundaries. Their study further supported their previous research, as they 
argue that the results suggest, again, that even at the sentence level, it is “the conjoint effective use of 
processes and storage” which causes individual differences in working memory abilities (Daneman & 
Carpenter, 1983). 

An fMRI study by Oleson, Westerberg, and Klingberg (2003) is more encouraging to working memory 
intervention research. They report that, following a 5 week training on working memory tasks, fMRI results 
in healthy, adult human subjects show an increase in brain activity in the middle frontal gyrus and superior 
and inferior parietal cortices, areas of the brain related to working memory. Oleson (2003) concludes from 
these results that, although “working memory capacity has traditionally thought to be constant,” it can be 
improved by training. 

While Swanson, Montgomery, and Oleson offer research supporting working memory interventions, 
Melby-Lervag and Hulme (2012) conducted a meta-analysis of the effectiveness of working memory training 
and concluded that, while the current literature reveals improvements in working memory, these 
improvements are restricted to the tasks practiced during training and not generalized to other tasks requiring 
working memory, like reading comprehension. 

Because there still exists conflicting theories regarding the possible benefits of working memory training, 
and the dearth of research utilizing an intervention based on sentence -level comprehension, this study will 
utilize a training task which specifically addresses the comprehension of sentences, in the hopes that it will 
not only improve working memory function, but will address a comprehension component as well. 
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One possibility is that such training can encourage development of processing strategies that enhance 
comprehension performance by utilizing event templates in long-term memory, while freeing up the working 
memory capacity. Earlier fMRI and EEG studies (Newman et ah, 2013; Malaia et ah, 2009) showed that 
individuals vary with regard to frequency and timing of attempts to consolidate incoming information in the 
working memory with existing event representations, or templates, in the long-term memory. Prolonged and 
repeated use of these strategies can lead to neural changes that enhance both online comprehension and 
recollection of content in reading (Newman & Malaia, 2013). 

Research in working memory indicates that working memory capacity is related to reading 
comprehension, but that research on the affect of interventions to improve working memory capacity has 
proven to generalize to complex working memory tasks. To further explore these findings, the research 
question is: Can reading comprehension be improved by strategic updating of WMC and utilization of 
episodic memory during reading? The researcher’s hypothesis is that a carefully designed reading 
comprehension intervention which specifically trains working memory in processing complex sentences will 
result in gains in working memory capacity as measured by a backward span task. Gains in reading 
comprehension of extended narrative text are beyond the scope of the current project. 

Teachers, parents, and school administrators interested in the reading success of students would be 
interested in an intervention that meets a specific and diagnosed need, such as low reading comprehension 
paired with identified poor working memory capacity. 


2. METHODOLOGY 

Because a predominance of literature addressing working memory capacity supports the notion that working 
memory capacity is static, this study attempts to address the issue from the stance of strategic use of working 
memory by providing an intervention for elementary students low working memory capacity, and explore 
whether this training will generalize to affect the results of a number -related working memory task 

2.1 Participants 

The study tested 11 fifth grade students (age M=10.9, SD=0.83, 7 females) attending a privately funded 
school, the primary function of which is to serve children from low-income households. All participants 
competed informed assent forms approved by the IRB of University of Texas at Arlington; consent was also 
obtain from each student’s parent or guardian. 

2.2 Experimental Design 

All participants completed two tasks: a backward span task to determine working memory capacity, and a 
baseline sentence comprehension task. The backward span task measures the working memory capacity of 
the participants by asking them to recall an increasingly larger set of numbers from end to beginning. The 
sentence comprehension task was designed to measure the participant’s ability to comprehend sentences as 
the length of each sentence become longer, increasingly taxing working memory capacity. The backward 
span task was performed until the participant reached failure. The sentence comprehension task was 
comprised of 18 sentences of varying lengths. 

2.3 Backward Span Task 

Beginning with a series of two one-digit numbers and increasing until the unable to recall in reverse order the 
series of numbers, the subject is asked to listen to the series of numbers, and then restate them in reverse 
order until an upper limit of correct recall is reached (see Eigure 1). 
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2 digits 4 8 

2 5 

3 digits 3 7 2 

5 8 6 

4 digits 2 8 9 3 

0 8 5 9 

5 digits 4 7 3 2 1 

6 3 15 2 

6 digits 6 0 9 3 4 8 

8 5 6 9 2 4 

Figure 1. The Backward Span Task 


2.4 Sentence Comprehension Task 

The sentence comprehension task consisted of a randomly presented sentences ranging in length from 5 to 19 
words. A memory probe element was included at either the beginning, middle, or end of the sentence. After 
reading the sentence aloud, the participant viewed 2 pictures and chose the one that best matched the 
meaning of the sentence (see Figure 2). 



2. The raccoons watched me paint the fence, (beginning) 
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Eigure 2. Sample Intervention Sentences 


2.5 Task and General Procedure 

1 1 fifth grade students from a self-contained classroom performed a baseline sentence comprehension task, 
Backward Digit Span pre-test, and a pseudoword reading task. One participant dropped out of the study after 
pre-testing. The remaining participants were divided into two groups of 5 students each, the experimental 
(those receiving the intervention, age M=10.8, 5D=0.84, 3 females) and the control (those not receiving the 
intervention; M=10.6, SD=0.89, 4 females). 

The 5 participants in the control group then participated in an intervention consisting of a working 
memory training two days a week for four weeks. The intervention consisted of students reading aloud 
sentences, and then identifying a picture which most accurately illustrated the sentence (see fig. 2). During 
the course of the intervention the sentences were gradually lengthened to 31 words. Following the 
intervention, both groups performed the Backward Digit Span; each participant also completed an alternate 
version of the sentence comprehension tasks in a post-test procedure (the versions administered as pre- and 
post-tests were balanced across participants). 

2.6 Data Analysis 

Accuracy and response time from sentences in pre- and post-intervention testing, and backward digit span 
were analyzed using a between-subject ANOVA with factor (Pre/Post). 


3. RESULTS 

A between-group repeated measures ANOVA analysis of Pre/Post measures (accuracy and response time for 
all sentences/sentences with probe words at the beginning, middle, and end/sentence lengths of 5, 7, 9, 1 1, 15 
and 19 words; Backwards Digit Span) revealed a Pre/Post x Group interaction, showing that the response 
time of the intervention group was significantly lowered on sentences with the comprehension probe 
appearing in the middle of the sentence (F (8,1)= 7.329, p<.027; from M=3155 msec, SD=353 msec, to 
M=2355 msec, SD 357 msec in intervention group, compared to non-intervention group pre-test RT M 
=2333 msec, SD=353 msec, and post-test M=2434, SD 357 msec), and for 5-word sentences (F(8,l)=4.874, 
p<.058, from M=3197 msec, SD=415 msec, to M=2136 msec, SD 153 msec in intervention group, compared 
to non-intervention group pre-test RT M =2224 msec, SD=415 msec, and post-test M=2340, SD 153 msec). 
These results suggest improved automaticity of access to short-term memory in the intervention group. 

Additionally, pre- and post-intervention Backward Digit Span Task results decreased for the intervention 
group (F(8,l)=8.333, p<.02; intervention group, from M =3, SD = 0.34, to M=2.6, SD=0.4; control group, 
from M=3.2, SD=0.34, to M=4.6, SD=0.4). Possibly, this indicates less resource allocation to the executive 
component controlling working memory manipulation as a result of higher resource allocation to the 
executive component controlling recall (as indicated by the response time results above). No other significant 
effects or interactions were found (p>.05). 


7 



ISBN: 978-989-8533-18-0 © 2013 lADIS 


4. DISCUSSION 

The research question asked whether the training in consolidation of verbal working memory content to 
long-term memory reference would improve performance on a general (numerical) working memory task. 
The data did not support the idea that verbal working memory consolidation training would improve working 
memory more generally, as measured by the Backwards Span Task. However, analysis of the pre- and 
post-test data taken from the Sentence Comprehension Task did indicate improvement in the intervention 
group participants’ performance in three ways: 1) participants more accurately recalled sentence elements 
that occured in the middle of the sentence, 2) participants processed sentence meaning in 5-word sentences 
more quickly as evidenced by faster response times, and 3) participants appeared to lower resource allocation 
to working memory manipulation, as opposed to recall and maintenance. 

A limitation to this study was its small sample size. A priori power analysis indicated that this study 
needs 24 subjects, 12 in each groups, to register a large size effect of the intervention (f-.4).when employing 
the traditional .05 criterion of statistical significance. In addition, for 5* grade students, the Sentence 
Comprehension Task (pre- and post-test) might have been more sensitive with sentences longer than 15 
words; so it is possible that the training task as it is would be better suited to younger participants. 


5. CONCLUSION 

This study is the first one to indicate that training in consolidating verbal information can improve individual 
performance on complex tasks requiring verbal working memory updates. While response time (indicating 
automaticity of the performance) was improved in the intervention group, performance on the Backward 
Digit Span task was lower post-intervention. It is not clear whether the training allowed the participants 
increased executive control of their working memory resource allocation, or improved the ability to switch 
attentional resources from working memory to long-term (semantic) memory. Overall, however, the results 
of this study indicate that it is possible to enhance individual executive component of working memory by 
short-term targeted training, and could lead to the development of materials for teachers to improve student 
reading comprehension at the sentence level. 
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ABSTRACT 

In this paper e-learning is used as an umbrella term for all types of learning involving technology. Graesser et al (2007) 
note that technologies for learning exist that allow for e-leaming systems to be much more than information delivery 
systems, but "unfortunately, the learning strategies of most students are extremely limited, so the systems must provide 
modeling of effective strategies, intelligent scaffolding, and accurate feedback" (p. 211). Self-efficacy is one area of 
human functioning where well-designed e-learning systems may be able to enhance performance. This paper was written 
to propose the intentional application of established instructional design practices and learning theory concepts for the 
purpose of creating e-learning environments that support the development of positive self-efficacy beliefs. Positive self- 
efficacy beliefs should, in turn, lead to enhanced achievement. 


KEYWORDS 

Self-efficacy, e-leaming, self-regulation, instmctional design 


1. INTRODUCTION 

E-learning is a term that has become an umbrella term for all types of learning involving technology 
(Dempsey & Van Eck, 2012). Graesser et al (2007) categorize many classes of technologies used in 
e-learning systems as computer-mediated technologies and note that "unfortunately, the learning strategies of 
most students are extremely limited, so the systems must provide modeling of effective strategies, intelligent 
scaffolding, and accurate feedback" (p. 211). One area of human functioning where well-designed e-learning 
systems may be able to enhance performance is self-efficacy. 

Self-efficacy refers to “beliefs in one’s capabilities to organize and execute the eourses of action required 
to produce given attainments” (Bandura, 1997, p. 3). Academic self-efficacy refers to those beliefs in the 
context of academic performance. Self-efficacy research in the context of learning has a long and rich body 
of research, in which many subject areas, learner groups, and delivery modes have been investigated. Pajares 
(2007) observed that “findings have now confirmed that students’ academic self-efficacy beliefs powerfully 
influence their academic attainments independent of possessed knowledge and skills, and that self-efficacy 
mediates the effect of such knowledge, skills, or other motivational factors” (p. 115). Learners with positive 
self-efficacy beliefs are more likely to participate, engage, persist, and have fewer negative emotional 
reactions in learning environments than students who lack these self-efficacy beliefs (Zimmerman, 2000). 

This paper was written to propose the intentional application of established instructional design practices 
and learning theory concepts for the purpose of creating e-learning environments that support the 
development of positive self-efficacy beliefs. The creation of learning environments that are structured 
specifically to support the development of positive self-efficacy beliefs, while important for all content areas, 
may be especially important in content areas where learners are traditionally less than optimistic about their 
ability to be successful (e.g. science or mathematics). 
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2. BODY OF PAPER 

In the last thirty years, research has provided ample support for the assertion that learner achievement is 
positively correlated with learner self-efficacy. Zimmerman (2000) summarized that “there is evidence 
(Bandura, 1997) that self-efficacious students participate more readily, work harder, persist longer, and have 
fewer adverse emotional reactions when they encounter difficulties than do those who doubt their 
capabilities” (p. 86). Four sources through which an individual develops self-efficacy beliefs are traditionally 
proposed: mastery experiences, vicarious experiences, verbal persuasion, and physiological and affective 
states (Bandura, 1997). Schunk and Pajares (2002) identified other sources of self-efficacy development, but 
commented that at least some of those sources are combinations of the four sources identified by Bandura. 
Some (e.g. Usher, 2009) have suggested that Bandura’s list of four sources is not exhaustive and not 
necessarily in order of power of influence. However, these four sources are often cited as the major sources 
of self-efficacy development, and they will be used as the anchoring points of the present paper. A brief 
description of these four sources begins with mastery experiences. 

2.1 The Sources of Self-Efficacy 

Mastery experiences refers to the learner's prior success, or lack of success, at a given task. Prior success at a 
task, or a similar task, should provide the learner with positive beliefs about the ability to be successful with 
the task at hand. Likewise, a lack of prior success, either because the task is perceived as totally new or 
because of previous failure, will not enhance, and may reduce a learner's belief regarding success with the 
task at hand. Vicarious experience impacts a learner's self-efficacy beliefs for a task based on perceptions 
formed from observations or knowledge of others performing the same, or similar tasks. The more closely the 
observed individual is perceived as comparable to the observer, the more the observer's self-efficacy is likely 
to be impacted. Verbal persuasion refers to feedback the learner receives from others regarding their 
perceived belief in the learner's success at a specific task. Many factors contribute the strength of the ability 
for the feedback to enhance one's self-efficacy. For instance, the perceived credibility of the persuader is 
important. Finally, physiological and affective states contribute to self-efficacy beliefs. These elements are 
"especially relevant in domains that involve physical accomplishments, health functioning, and coping with 
stressors" (Bandura, 1997, p. 106). This area may not appear to be directly link to the design of e-learning 
environments, but there are many instances where e-learning systems can contribute to user stress. 
Therefore, design elements to reduce user stress may have benefits where self-efficacy is concerned. Let us 
now consider how the sources of self-efficacy can be addressed in e-learning environment design. 

2.2 Sources of Self-Efficacy and e-Learning Environment Design 

2.2.1 Scaffolding Recognition of Mastery Experiences in e-Learning 

As observed by Hodges and Murphy (2009), connections between prior experiences and new experiences in 
e-learning environments, may not be readily apparent to some learners. In those situations, the importance of 
mastery experiences in the development of learner self-efficacy may be lessened because the learner does not 
realize them. To counteract the possible overshadowing of the new e-learning environment, care must be 
taken to alert learners to prior successes. Possible strategies for this include reminding the learner to reflect 
on past success. It may be important to stress prior success with both content and delivery mode. This could 
be accomplished by prompts asking the learners to make these reflections, or by asking learners to complete a 
pre-course survey to collect information about prior successes, then simply reminding the learner that success 
with the content, or in e-learning experiences, has been achieved. Education providers are collecting 
increasing quantities of information on learners, and information on relevant prior successes could be 
leveraged to provide the information needed to remind the learners about those successes. This type of 
intelligence within a system is preferred so that students are not prompted to recall a lack of prior success, 
which may decrease their self-efficacy. From a design standpoint, knowledge of a lack of prior success could 
be utilized by the e-learning system to customize course offerings and features in ways to enhance 
self-efficacy. For example, it may be better, from a self-efficacy standpoint, for some learners to be 
presented with course material in smaller “chunks” (Dick & Carey, 1996). 
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Small chunks may give them an opportunity to achieve success, thus creating mastery experiences that 
can enhance self-efficacy. Also, as the course progresses, learners could be routinely reminded of recent 
successes in the course. These routine reminders could be automated and might take many forms, perhaps 
depending on user set preferences. A learner might get an email message congratulating him or her on a 
recent high level of success on a quiz or test, or some type of on-screen badge might be selected as the 
desired form of validation of academic success. 

2.2.2 Scaffolding Vicarious Experience in e-Learning 

Learners in many e-learning environments are isolated from each other (e.g. Park, 2008, p.l6). This is in 
many instances an artifact of the asynchronous, any time, anywhere selling point of many e-learning 
programs and not a specifically desired design feature. If learners are not in situations where they can 
physically see peers, or anyone associated with their e-learning experience, then designers should consider 
how vicarious experience could be scaffolded to assist with the development of positive self-efficacy beliefs. 
Graesser et al note that animated pedagogical agents “can mimic face-to-face communication with human 
tutors, instructors, mentors, peers, or people who serve other roles” (p. 217). Pedagogical agents in the 
mentor or peer role may be useful aids to vicarious experience related to self-efficacy development. Some 
work in this area has already been documented (e.g. Baylor & Kim, 2005). Less fantastic approaches to the 
vicarious experience component of self-efficacy development include testimonials from former, successful 
learners and the publication of aggregate performance data for similar learners. Schunk (1991) notes that 
“observing similar peers perform a task conveys to observers that they too are capable of accomplishing it” 
(p. 208). In many e-learning environments the direct observation is not possible. Publishing performance data 
on assignments from previous instances of the class, or live data from a current class for similar students 
would allow students in isolated e-learning environments to see how peers are performing. Would knowledge 
that other students are performing better or worse than an individual prompt a student to change study habits 
or seek assistance? How would knowledge of this type impact learner self-efficacy and achievement in the 
class? An important area of inquiry around these questions would determine the levels of peer achievement 
that serve to enhance self-efficacy in productive ways. For example, a student that sees he or she is doing 
much better than the peers in the classroom, may become over confident resulting in eventually diminished 
performance. On the other hand, a student that is doing much worse than his or her peers may suffer 
decreased self-efficacy, negatively impacting achievement. A key practice to avoiding these types of errors 
would be defining what is meant by the term peer so that students are compared only to those fellow students 
for whom there is a reasonable expectation of similar performance. 

2.2.3 Verbal Persuasion in e-Learning 

Verbal persuasion, perhaps more appropriately referred to in e-learning contexts as “social persuasion” 
(Bandura, 1997, p. 101), can take many forms. Feedback from instructors is probably the most frequently 
thought of example of social persuasion. However, informal feedback from classmates, or automated 
feedback from grading systems in the e-learning environment may be useful for enhancing learner self- 
efficacy. Pedagogical agents may be leveraged for the purpose of social persuasion and its influence on 
learner self-efficacy. 

In the context of e-learning systems, there may be an interaction between feedback that combines the 
vicarious experience and social persuasion aspects of self-efficacy development. For example, feedback 
about a learner's ability to succeed because other similar students are succeeding straddles both of these 
categories. Feedback to learners is an important element of many instructional and learning theories. The 
social persuasion source of self-efficacy development is the source most closely associated with direct 
feedback to learners and, therefore, it should receive some significant attention in the design of e-learning 
systems. The development of useful, scalable technologies that address social persuasion may be even more 
important as e-learning systems increase in use and expand into large-enrollment systems like those planned 
for massive open online courses (MOOCs). 

Email has been the targeted technology used by some researchers investigating verbal persuasion. 
Jackson (2002) used email messages that emphasized past successes of learners, related the fact that similar 
learners had previously achieved success, encouraged learners to work hard and stay on task, and provided 
stress-reduction tips. 
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Self-efficacy was significantly related to performance, and self-efficacy was enhanced in Jackson’s 
experiment. Hodges (2008) found that specially designed email messages related to prior achievement did 
not enhance self-efficacy, but self-efficacy was positively related to achievement. 

As learners’ preferred modes of communication change, and features of learning systems evolve, 
attention will need to be given to the way such messages are delivered. Will learners want this type of 
feedback to be provided only within the learning system, or will they prefer the information be sent outside 
of the learning system? Who is best to deliver this type of information? Bandura (1997) highlights the 
importance of the source of the social persuasion and the source’s perceived credibility. Investigations into 
how the credibility, influence, and success of the persuader are effected by the technology used to deliver the 
persuasive message will be important as more intelligent and automated systems are constructed. 

2.2.4 Physiological and Affective State in e-Learning 

Bandura (1997) explains the relationship between behavior, personal factors, and environmental factors as 
triadic reciprocalism (p. 5-6). The physiological and affective state of learners in e-learning environments can 
be addressed by taking efforts to create environments that are non-threatening and comforting, not frustrating 
to use, etc. This includes attention to message design, usability, and accessibility. Some learners are 
exceptionally stressed with respect to certain topics (e.g. math anxiety). In these cases, care must be taken to 
design environments that do not contribute to the learners' stress. For example, on screen timers measuring 
the length of time spent, or remaining, to work problems may be viewed as stressful for some learners. 
Designers of e-learning environments may not have control over the environments where their learners are 
using the system, but learners should be prompted to consider their physiological and affective states when 
they access such systems. It may be wise to take a break and return to the system when they are more 
prepared to engage with the system effectively from physiological or affective perspectives. 

All learning systems should be subjected to intense usability testing so ensure that the act of interacting 
with the system is not somehow negatively impacting the learner. In the absence of time or resources for 
extensive usability testing, learning systems should be reviewed carefully against accepted best practices. 
For example, the Quality Matters (2013) initiative has published guidelines that stress organized materials 
and ease of navigation in online courses. 

Table 1 summarizes how the traditional four sources of self-efficacy might be addressed in learning 
systems. 

Table 1. Possible Self-efficacy Strategies to Incorporate in Learning Systems 


Self-efficacy source 

Suggested learning system feature 

Mastery experiences 

Providing information from academic history about 
former, related successes; automated messages that 
highlight recent prior success 

Vicarious experience 

Video testimonials from former, successful students; 
provide aggregate peer data to show relative 
performance 

VerbaFsocial persuasion 

Messages delivered by the learning system 

PhysiologicaFaffective state 

Usability testing; careful review of navigation and 
organization 


2.3 Self-efficacy and Self-regulation 

Zimmerman (2002) defines self-regulation as the "self-directive process by which learners transform their 
mental abilities into academic skills" (p. 64). Self-efficacy beliefs and the ability to self -regulate learning are 
strongly linked together. Students who can self-regulate their learning are able to achieve and realize 
academic successes, which in turn enhances self-efficacy for that particular learning task. As self-efficacy 
increases, the learner is more apt to persist at a skill, applying self -regulation strategies to continue with the 
learning task. The effective application of self-regulation strategies is essential for success in academic 
endeavors and the development of self-efficacy (Pajares, 2007, p. 1 19). 
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Therefore, any e-learning system designed and built on a self-efficacy framework, must include support 
for self-regulatory strategies. The literature base for academic self-regulation is well developed, and several 
important self-regulation strategies have been identified. Many of these strategies can be supported or 
encouraged through features in e-learning systems. For example, Pajares (2007) lists “finishing homework 
and assignments by deadlines” (p. 119), “using the library for information for elass assignments” (p. 119), 
and “participating in class discussions” (p. 119) as identified self-regulatory practices. Each of these practices 
can be supported through features built into e-learning systems. 

2.4 Learning Analytics 

Some of the features and strategies suggested in this paper require that the e-learning system have access to 
various demographic, preference, and performance data of the learners using the system. Access to, and use 
of, this type of data is gaining interest in the Education community. Johnson et al define Learning Analytics 
(LA) as “an emergent field of research that aspires to use data analysis to inform decisions made on every tier 
of the educational system” (p. 24) and note that it is an important upcoming technology for teaching and 
learning. The use of large data sets for decision making is not a new idea, but its application to educational 
endeavors is relatively new in the sense that tools and businesses are offering LA to a broader Education 
audience. Roy Pea (2013) has promoted combining LA tools and techniques with the theory and research 
from the field of Learning Sciences to personalize learning on large scales. Some of this personalization 
could be aimed at enhancing the self-efficacy of learners in e-learning environments. Not unlike current 
music and retail analytics that suggest new songs or artists, or other products a consumer may like, LA could 
be used to make suggestions about when learners need to seek assistance or change study habits. LA could be 
utilized to customize the e-learning environment, as described in this paper, to facilitate mastery experiences 
by helping learners recall past success or customizing how content is chunked; creating personally 
meaningful pedagogical agents to aid with vicarious experiences and/or social persuasion; and informing 
learners of the performance of other learners in the same e-learning environment. LA also may aid in the 
formation of positive self-efficacy beliefs through a combination of self-regulation and social persuasion or 
vicarious experiences. Lor instance, LA may be able to identify trends in successful students related to 
discussion participation or engagement with e-learning system features such as a calendar of due dates. The 
system could then inform learners about the behaviors of successful peers. 

2.5 Professional Development for Course Facilitators 

Many of the design features suggested thus far are possible in modern learning management systems at some, 
less than automatic level. For example, the current version of Desire2Leam’s learning management system 
includes intelligent agents that allow the course facilitator to send messages to targeted groups of learners. 
The granularity with which learners can be targeted is small. This type of targeting messaging could be used 
as a middle-ground to a fully automated system for delivering peer information to influence vicarious 
components of self-efficacy development, or for delivering instructor feedback to address issues of social 
persuasion. Also, some unused features of learning management systems often can be hidden from learners, 
thus simplifying the screens used and increasing the usability of the system. There is often no shortage of 
workshops or other learning opportunities for course facilitators to learn basic learning management skills 
such as how to use the built-in system grade book. However, the rationale for the use of various features from 
a psychological perspective is often not included in these workshops. More professional development is 
needed that provides course facilitators with an explanation of why some features of a learning management 
system should be used from a learning design perspective. 


3. CONCLUSION 

One may notice that the suggestions in this paper share some of the same elements as Keller’s (1987) ARCS 
model, however, there are differences. Keller’s seminal work was a synthesis of many motivation concepts, 
but the current paper is focusing on learner self-efficacy. 
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Self-efficacy falls into Keller’s confidence component of the ARCS model, but self-efficacy and 
confidence are not the same (see Bandura, 1997, p. 382), hence the application of Keller’s work may aid in 
the development of positive self-efficacy beliefs, but perhaps not with the same amount of focus as the 
suggestions included in the present paper. This paper also includes ideas for integrating current technologies, 
such as Learning Analytics, to influence self-efficacy in ways that were not possible when ARCS was 
emerging. 

Many of the design features of e-learning systems suggested in this paper have been implemented or 
studied in isolation. In this paper they have been collected together and organized around self-efficacy 
theory, with the primary purpose of suggesting that e-learning systems be designed to enhance learner self- 
efficacy. The long history of self-efficacy theory in the research literature, and the positive relationship that 
has been demonstrated between academic self-efficacy and learner achievement, provide a strong rationale 
for designing and developing an e-learning environment with this focus. Note that the ideas suggested in this 
paper are simply examples of how the various sources of self-efficacy could be addressed in e-learning 
environments. The list of ideas and examples is not meant to be exhaustive. It is intended to provide 
suggestions of areas where existing or emerging learning technologies can be applied toward the goal of 
enhancing learner self-efficacy. There are many ways in which strategies for enhancing self-efficacy may be 
interpreted and implemented in e-learning environments. 
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ABSTRACT 

‘Digital Native’ and ‘Digital Immigrant’ are terms, popularized by Prensky (2001), to describe those bom either before, 
or in the digital era (i.e. after 1980). In recent years, this dichotomy has been used to raise awareness of differences in 
technology usage and what these differences may mean for education. The present study examines Japanese university 
teacher and student use of digital technology in academic settings, as well as their preferences for digital or paper-hased 
educational practices. At a private university in Japan, 337 first-year university students and 170 full-time faculty 
responded to tailored questionnaires addressing digital technology use. Both questionnaires focused on four areas: 
technology use, self-efficacy, familiarity, and leaming/teaching style preferences. The results showed that there are clear 
differences in technology use between teachers and students, with teachers using computers more than students. 
Self-efficacy of students was much lower than teachers, and there were distinct differences in the familiarity with 
software and web pages between the two groups, with students being much more comfortable with websites and teachers 
being more familiar with software. Learning/teaching styles of both groups were similar, with a bias towards teacher- 
centered and paper-based educational practices. These findings suggest that there is a mismatch between institutional 
goals and the beliefs and practices of both teachers and students regarding the role of technology in the curriculum. 


KEYWORDS 

Digital literacy, student attitudes, teacher attitudes, digital native. 


1. INTRODUCTION 

Recently, particularly in the US and Europe, educators are arguing that a new generation of learners is 
entering higher education, one which has grown up with digital media and technology as an integral part of 
their everyday lives. It is claimed that this group’s use of information and communication technology (ICT) 
sets them apart from previous generations of students and, in particular, from their teachers. It has been 
argued that the importance of new technologies within the lives of these young people is so significant, that 
how and what is taught must fundamentally change to adapt to the new skills and cognitive and social 
features of these ‘digital natives’ (Prensky, 2001a; Gibbons, 2007; Underwood, 2007). At the same time, 
many institutions are rapidly moving towards more digitally based learning, with CALL curriculum and CMS 
and LMS courseware becoming more and more common. Although the push for more digitally-based 
learning, and the use of digital technology can be seen in many parts of the world, it should be argued that 
major changes in teaching and the educational environment need to be carefully balanced with the results of 
empirical research. 

In this paper, we explore this topic in terms of Japanese higher education, providing data in an effort to 
clearly understand the current state of teachers and students in the Japanese educational system. In addition, 
we offer a critique against the common conception of the digital native as based purely on generational 
differences, and show how a clearer understanding of teachers’ and students’ use of digital technology can be 
of help to both educators and administrators in higher education. 
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2. REVIEW OE THE LITERATURE 

2.1 Digital Native Definition 

Central to the original definition of the digital native is that people born in the last two decades have always 
been surrounded by, and interacted with, digital technologies. 

Prensky refers to people who were born before 1980, as ‘digital immigrants’, and suggests that they may 
adopt new technologies but will still have strong connections to the pre -digital past, making them unable to 
fully understand the natives. 

Prensky (2001a, 2001b) argues that one of the consequences of the digital environment is the way that 
young people think and process information compared to older generations. He argues that this younger 
generation is used to fast reception and transmission of information, allowing them to parallel process and 
multi-task. Being brought up on video, DVDs, and gaming, they prefer graphics before text. They expect 
instant gratification and frequent rewards, as can be seen in video games. 

Jukes and Dosaj (2006) have created descriptions of behaviors that they feel differentiate digital native 
learners from many of their teachers. They argue that digital immigrant teachers prefer a slow and controlled 
release of information, singular tasking (as opposed to multi-tasking), text over pictures, sound and video, 
linear and sequential presentation, standardized testing, and delayed rewards. Digital native students, on the 
other hand, prefer multimedia, parallel processing and multitasking, pictures and video before text, 
immediate relevance, and instant gratification. 

This definition then suggests that the differences between digital natives and the digital immigrant groups 
have profound implications for education: if young people now have a range of different preferences that do 
not match current educational practices, then the current pedagogies need to change. In fact, many schools 
and teachers have not responded to the alleged new ways in which students communicate and access 
information. Although ICT has allowed university educators to create environments that are more closely 
aligned with the preferences of digital natives, one downside seen in the US is a gap or ‘digital disconnect’ 
between students and teachers (Tapscott, 1997; Underwood, 2007). Teachers and administrators set the tone 
for ICT use at school, resulting in a gap between how teachers approach ICT use and how students approach 
it. Prensky (2001a) argues that gap has resulted in the digital natives being taught by digital immigrants who 
are not talking the same language. 

But is this actually the case? Those in support of this digital native/immigrant divide usually assign broad 
characteristics (e.g. a specific learning style or set of learning preferences as listed above) to an entire 
generation, and suggest all young people are technology experts (Bennett, Maton, & Kervin, 2008). 
However, there seems to be significant variation within the young population in how and why they use these 
new technologies and the effectiveness of that use (e.g. DiMaggio & Hargittai, 2001; Facer & Furlong 2001; 
Livingstone & Helsper, 2007). Helsper and Eynon (2010) support this argument with a large-scale survey 
that suggests not only age, but educational level, experience and gender also play a part. These last two 
studies suggest that if there is indeed a gap between students and teachers, that it is possible to close it. This 
is an important point, as the majority of evidence in support of the concept of the digital native is based on 
data related to age and specific use. 

2.2 Digital Native Research 

Although there are many studies supporting the idea of a digital divide (Kvavik, Caruso & Morgan, 2004), 
others suggest that young people are far from comfortable with all forms of digital technology (Kennedy, 
Krause, Judd, Churchward & Gray, 2006). In China, Li and Raniery (2010) reported that ‘digital natives’ in 
China are not necessarily digitally competent, with competency depending on educational background (the 
kind of school) and age. Interestingly, competency was not significantly influenced by PC ownership or 
Internet access/use. 

It must be noted that general ICT use may differ significantly from ICT use for learning purposes. Student 
use or non-use of technology for learning may be a complex relationship between competency, perceived 
usefulness, and support. In an effort to uncover factors that predict student use of technology for learning, 
Lai, Wang and Lei (2012) surveyed Hong Kong undergraduates regarding their ICT use and attitudes towards 
the use of technology in learning. 
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The results of their survey study suggested that compatibility between student learning styles and 
technology, support from peers and teachers, and general attitudes toward technology were predictors of ICT 
use for learning. In contrast, the students’ perception of their own ICT skills and the perceived usefulness of 
the technology were weaker predictors of student technology use. 

2.3 Teachers and Technology Acceptance 

Studies on technology acceptance have proliferated in recent years, with a wide variety of theoretical models 
being proposed to explain differences in individual acceptance and use of ICT. The technology acceptance 
model (TAM) promoted by Venkatesh et al. (2003) has proven useful in predicting adoption and use of ICT. 
By investigating perceived usefulness, performance and outcome expectancies, and relative advantage, user 
attitudes towards technology can be used to help explain acceptance and use of ICT. 

There seem to be some clear differences between teachers and students regarding technology, and this 
may lie in how they use the technology and how important they perceive it to be. Gu, Zhu, and Guo (2013) 
found that Chinese students had a longer experience using ICT than their teachers, with teachers using ICT 
more inside the classroom than outside. Their findings revealed a complex relationship between personal 
factors and social influence that explained teacher’s ICT adoption. In contrast, Ertmer et al. (2012) found that 
teachers’ pedagogical beliefs (i.e. student-centered or teacher-centered pedagogy) as well as their beliefs and 
attitudes about the relevance of technology in education played key roles in shaping ICT practices. The 
results of Ertmer et al. (2012) are similar to the reports of Li and Ni (2011). This survey of Chinese English 
teachers found that although the teachers held positive attitudes toward technology in education, they used 
technology mainly for teacher-centered purposes. One of the possible explanations given for this result was 
the predominance of teacher-centered practices and a lack of professional development. 

It may be that there is as much variation within the digital native generation as between the generations, 
and that differences exist across cultures as well. The purpose of the present study is to survey the current 
state of ICT use among Japanese university students, their learning style preferences, and compare these 
results with the results of a similar study with their teachers. If there is indeed a distinct digital native 
generation in Japan, and their learning style preferences match the model hypothesized by Prensky (2001) 
and Brown (2000), then changes to the teaching style and curriculum of higher education are warranted. 

The present study seeks to understand student and teacher attitudes towards digital technology. It is hoped 
that through understanding the ‘digital state’ of our students and teachers, we can better inform teachers and 
administrators on how best to implement ICT in the university environment. Two questionnaires were 
devised to address the following general questions: 

• How does a sample of Japanese university students use ICT technology in academic and 
non-academic settings? 

• How does a sample of Japanese university teachers use ICT technology in an academic setting? 

• Do student learning style preferences lean more towards digital media, or paper-based media? 

• Do teachers’ educational style preferences lean more towards digital media, or paper-based media? 


3. METHODOLGY 
3.1 The Setting 

The university in question is a private university in Kyoto Japan, with over 13,000 students enrolled. The 
majority of the students are what can be considered social studies and liberal arts students, with a small group 
of science majors. The university itself is ranked in the top 10 in the area in terms of academic standing. In an 
effort to maintain that standing and attract more students in a country with a declining university student 
population, the university has been aggressive about modem infrastructure on campus. The campus has Wi-Ei 
throughout the buildings and in many public areas, has about 2,000 computers available to students in a 
variety of open computing rooms, library access, and CALL classrooms and computer labs. Eulltime faculty 
are provided with computers in their offices, and both faculty and staff have dedicated server space available 
to them for storing documents on the university server. 
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In addition, the university has adopted the Moodle CMS, both for students, faculty, and staff. The Moodle 
is used as the primary method of communication between administration, faculty, and the students. All 
courses have a dedicated Moodle page which can be accessed by both students enrolled in the course and 
their teachers. 

3.2 The Participants 

The data was collected from 337 first-year students, in eight faculties, in September, 2011. The respondents 
were predominantly male (74%) and age range was 18-19. Half of the students were living with their 
families, and half were living away from home (primarily in rented apartments). 

Data from 170 teachers (again, in eight faculties) was collected in September 2012. Respondents for the 
teacher survey were also predominantly male (81%), with an age range of 25-65. The teachers were all 
full-time faculty, and as such all had an office which could be equipped with a university-supplied computer. 

3.3 The Surveys 

The purpose of this study is to understand how technology is used and perceived by both university teachers 
and their students. Two survey instruments were used to collect data. The questionnaires examined: Japanese 
university students’ use of ICT in academic and non-academic settings, and their learning style preferences 
(digital- or paper-based); and Japanese university teachers’ use of ICT in academic settings, and their 
teaching style preferences (digital- or paper-based). The questionnaires covered four general areas: 
technology use, self-efficacy, familiarity, and learning/teaching style preferences. The student questionnaire 
consisted of 75 questions, and the teacher questionnaire consisted of 47 questions. The items in both 
instruments required responses on a four-point Likert scale. The student questionnaire was administered in 
class, in a paper-based format, and the teacher questionnaire was administered in a paper-based format via 
campus mail. The response rate for the teacher questionnaire was 47%. 


4. RESULTS 

The findings give an interesting view of the use of technology by these two groups in university settings, as 
shown below. All the results discussed here are based on the answers from the respondents from the 8 
faculties. 

4.1 Technology Use 

Figure 1 displays reported computer (PC) use by both teachers and students. Students seem to prefer using 
computers at home to using them at school. The majority of them use computers less than three hours a day 
(59% less than one hour, 29% 1-3 hours) while they use phones much longer (45% 1-3 hours, 36% more than 
3 hours). On the other hand, the majority of teachers (67%) used computers equally at home and at school, 
with the majority (74%) using them during daylight hours. Clearly there is a gap between when students 
access computers, with the students spending less time on computers in general, and very little time on 
computers while at school. 
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Eigure 1. Technology Use of Teachers and Students 


More specific information regarding activities for both groups can be found in Table 1 . This table shows 
the average ratings (on the four-point Likert scale: 1 being “use it a lot,” 2 “sometimes use it,” 3 “know about 
it, but don’t use it,” and 4 “don’t know it”) for each activity, comparing teacher activities with student 
activities. Although teachers were not asked how they used computers in non-academic settings, it is clear 
that they are using computers in a greater variety of ways than the students. 


Table 1 . Digital Activities 


Teachers 

Students 

Sending email and messages 

1.25 

Checking school info 

1.50 

Writing a report or paper 

1.32 

Research for a report 

1.55 

Creating instructional materials 

1.39 

Writing a report 

1.59 

Gathering information for research 

1.45 

Using bookmarks 

1.63 

Gathering information for planning lessons 

1.60 

Watching movies or TV 

1.66 

Keeping administrative records 

1.63 

Using a search engine 

1.80 

Checking school information (Post) 

1.64 

Listening to music 

1.81 

Presenting materials to class 

1.65 

Downloading music 

2.17 

Research for a report or paper 

1.73 

Online Shopping 

2.31 

Paying for goods 

2.08 

Social networking 

2.5 

Social networking (blogs, chatrooms) 

2.85 

Playing games 

2.59 



Editing photos and movies 

2.63 



Sending email 

2.69 



Reading books, magazines 

2.90 



Paying for goods 

3.01 


Note: scores closer to a value of 1 indicate greater usage 


4.2 Self-efficacy (Perceived Skill) 

Figure 2 reports the results of perceived computer skill. This was a general question asking the teachers and 
students to rate their general ability on a 5-point scale. The majority of the students (80%) rated their 
perceived computer skill as fair to poor, while the teachers rated their computer skills as good to fair (60%) 
with 15% of them rating their computer skills as excellent. This was a surprising finding, since it challenges 
the concept of the digital native and digital immigrant. 
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Figure 2. Perceived Skill 


4.3 Familiarity with Websites and Software 

The majority of the questions in the questionnaire were designed to investigate student and teacher 
familiarity with websites (Table 2) and software (Table 3). Concerning websites, only 4 items were marked as 
very familiar by the students (YouTube, Yahoo, Google and mixi*). Even though many of the students are 
required to use Moodle, Word, and Excel (and possibly PowerPoint) at school, their reported familiarity was 
not very high suggesting that their use of websites and software are very limited. The teachers had similar 
responses, though their familiarity with software was far greater than the students, and their use and 
familiarity with social sites such as Eacebook was much lower than the students’. 


Table 2. Familiarity with Websites and Software 


Websites 

Software 

Teachers 

Students 

Teachers 

Students 

Google 

1.60 

youtube 

1.38 

word processing 

1.30 

word 

2.11 

Yahoo 

2.21 

yahoo 

1.52 

presentation 

1.84 

iTunes 

2.15 

Moodle 

2.25 

google 

1.71 

spreadsheet 

1.99 

excel 

2.44 

Amazon 

2.34 

mixi 

1.87 

media players 

2.6 

skype 

2.46 

KSU-Cat 

2.54 

moodle 

2.10 

sound/movie editor 

2.75 

powerpoint 

2.58 

CiNii 

2.61 

amazon 

2.35 

statistics 

2.78 

mediaplayer 

2.76 

YouTube 

2.71 

rakuten 

2.59 

field specific 

2.80 

realplayer 

3.31 

Rakuten 

2.99 

twitter 

2.76 

online software 

3.01 

openoffice 

3.56 

Eacebook 

2.99 

facebook 

3.05 

communication 

3.05 

moviemaker 

3.56 

Twitter 

3.13 

excite 

3.52 



iMovie 

3.62 

Mixi 

3.25 

myspace 

3.64 



prezi 

3.63 



Flickr 

3.71 



googledocs 

3.66 



picasa 

3.75 






4.4 Learning/Teaching Preferences 

The last part of the questionnaire was about their learning and teaching preferences. We asked students they 
prefer to do school activities such as lectures and reading assignments in class or out of class, and paper- 
based or computer-based (Figure 3). Teachers were asked similar questions regarding their teaching and 
educational style (Figure 4). The student preferences seem to be quite traditional and are generally in class, 
paper-based. They prefer doing research and writing a paper outside of class, and computer-based, but for 
other activities, they showed preference to do them in class, and paper-based, especially taking a test/quiz and 
lecture. 


1 mixi is a social network site popular among Japanese youth. 
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Teachers were of a similar vein, with a preference for in-class lectures, quizzes and tests, and paper-based 
readings and materials. This is against the recent trend of the university administration which is quickly 
shifting from paper-based to computer-based distribution of information. 

Paper-based 

Reading 
Presentation 
Writing 
Research 
Test 
Quiz 
Lecture 

Eigure 3. Student Learning Preferences 


Eigure 4. Teaching Preferences 

5. CONCLUSION 

This paper examined the state of ICT and digital media use among a small sample of Japanese university 
students and their teachers, in an effort to determine the extent to which generation, experience in using the 
Internet, and breadth of use were indicators of digital nativeness. In contrast with the original digital native 
argument put forth by Prensky, it seems clear that generation or breadth of use alone does not define one as a 
digital native. It is obvious from the data that these students are exposed to and have a greater range of ICTs 
in their lives, but this does not guarantee confidence and skill in interacting with these technologies. The 
same goes for their teachers, who in many ways seemed to be more digitally active than their charges. 

There were two surprising findings from the present study. First was the self-reported general lack of 
computer use and skills among the majority of the students, and the contrary self-evaluation by the teachers. 
The second surprising finding is the preference for traditional forms of learning and studying in both the 
student and the teacher groups. These two general findings suggest that the students are not the digital natives 
that Prensky describes, and that the teachers are far from the digital immigrants that they are commonly made 
out to be. The results also parallel the findings of Ertmer et al. (2012) and Li and Ni (2011) suggesting that in 
this study as well, teacher beliefs and practices influenced the use of ICT in their teaching. 
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These findings then suggest that it would be a mistake for teachers and administrators to blindly adopt a 
digital learning and CMS route for all aspects of the university environment. The prevalent arguments for 
CALL and CMS — 24/7 access and flexibility, were not supported by the findings in this study. Whether 
accessible or flexible, this means little when the target users are ignoring the technology or prefer different 
media. Administrators and CMS providers clearly need to take student and teacher preferences for paper- 
based materials into account and not simply replace them with CALL alternatives. This is not to say that 
CALL and CMS should be avoided. Rather, a blended approach which combines both digital and paper-based 
materials should not be overlooked. 
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ABSTRACT 

Being aware of what is absent in faculty development programs is important for practitioners and researchers so that they 
can create and advance programs, and support the development of knowledge, skills, and abilities for distance education 
instructors. Requests for distance education courses persist to grow at the college and university levels, as students and 
corporations push for more flexibility in education. Administrators emphasize the return on investment, while faculty 
members develop distance education courses. More and more faculty members are being prompted to facilitate in the 
distance education and online arena. This paper focuses on the current state of knowledge needed by facilitators to teach 
and deliver education using androgogical techniques and technologies. Additionally it emphasizes the need to understand 
the different types of available technologies and understand their applicability to facilitate learning. These technologies 
are becoming more and more important to facilitators and administrators as the Internet continues to energize educational 
deliveries. 


KEYWORDS 

Distance Education, Performance, Technology, Andragogy 


1. INTRODUCTION 

Distance education and exploding technological progressions are reforming the way instructors, 
administrators, and students conduct educational actions and use technologies. These reformations of actions 
are driven by students’ and corporations’ push for flexibility in education, administrators’ focus on these 
demands and their return on investments, and faculty members’ being nudged to instruct in the distance 
education arena. This learning can be synchronous or asynchronous. Before moving forward lets define 
distance education. 

Defining a field of study is substantial as definitions indicate practice in that field and places information 
in proper context (Seels and Richey, 1994). Distance education is defined and termed in many ways. Defined 
by Yacci, “distance education is the practical subset of education that deals with instruction in which distance 
and time are the criterial [sic] attributes; that is, student and teacher (and other students) are separated by 
distance and/or time” (2000, p.l). “Online learning overlaps with the broader category of distance learning, 
which encompasses earlier technologies such as correspondence courses, educational television and 
videoconferencing” (U. S. Department of Education, 2010, p. xi). For the purposes of this paper distance 
education will be the term used to refer to for distance learning, elearning, ilearning, online learning, and co- 
learning. 

Wrapped in distance education and online learning is technological advancement. Facilitators must 
become comfortable in understanding the different types of technology and the best way to employ 
technologies in the learning environment. Technology not only helps develop learning, it bridges gaps in 
distances. January 2008 the University of Illinois Urbana Champaign implemented “the University of Illinois 
Global Campus - an integrated online programme [sic] created in collaboration with the colleges and 
academic departments at the university’s residential campuses” (Economist Intelligence Unit, 2008, p. 8). 
This implementation is an instance of the way distance education professionals located in the United States 
can connect with colleagues in different countries as if they are located in the same room. This connection is 
by using collaborative software applications that support the desktop view of a host computer to shared 
applications, voice, and video with numerous computers during conferencing. 
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Even though technology continues to electrify the educational arena, “...we are cognisant [sic] of the 
need to be critically selective and thoughtful about what technology is used and how it is used. (Latham and 
Carr, 2012, p. 41) 

The question here is, What is the present tactic to deliver facilitation training for faculty who teach 
onlinel According to Allen and Seaman “There is no single approach being taken by institutions in 
providing training for their teaching faculty. Most institutions use a combination of mentoring and training 
options,” (2011, p. 6). The subsequent question is What technologies should facilitators understand and 
employ when teaching onlinel The answers to these questions will be uncovered throughout this document. 

This paper contains foundational information needed to examine, illuminate, and comprehend the current 
state of problems (i.e., lack of understanding how to teach in the distance education environment), and the 
types of technologies (i.e., learning management systems, blogs, wikis, collaboration rooms, etc.). The need 
to better understand the needed educational model and which technologies offer facilitators advantages in 
delivering instruction through distance education will be explored. 


2. BACKGROUND OF THE PROBLEM 

As learning offered through distance education via the Internet is exploding, so is the directive for 
university/college faculty members who will facilitate these courses. Traditional academic expectations 
remain unchanged. The perceived notions of online education course deliveries taking more time than face- 
to-face endure to circulate (Drago, Peltier &Sorensen, 2002; Hinson LaPrairie, 2005; Allen & Seaman, 
201 1). These perceptions of more time being needed to teach online than face-to-face is noted in survey and 
qualitative interview studies (Van De Dord, Korolyn, 2012; Bolliger Waslik, 2009; Conciecao, 2006; 
Haber & Mills, 2008). 

Conversely, the current medium, teaching through distance education brings a fresh environment for 
faculty and administrators to live, work, and balance their personal and professional lives. Consequently, 
faculty and administrators must sincerely sort through educational lessons learned. Next they need to then 
merge this new understanding into today’s changing social technology driven society to preserve the online 
competitive edge. This can occur by faculty and administrators creating faculty development programs that 
are performance driven towards faculty being able to understand and use multiple technology modalities. 
Faculty will need to apply the usage of multiple technologies during courses. This technology tool knowledge 
needs to be developed into a strong distance education competency model and applied for all faculty and 
administrators. 

2.1 Present Tactics to Eacilitate through Distance Education 

Marginal numbers of facilitators are experienced teaching online. When reviewing facilitators with 
experience teaching online, numerous changes were documented regarding their experiences. Some changes 
comprise: Revising projects to drill down to their key points; these are examples of how assignments 
normally given in face-to-face settings need to be changed. Remaining in the face-to-face setting will not 
produce the same results online. (2) Next is to determine the fostering of collaboration between 
geographically dispersed individuals (Chronicle of Higher Education, 2010). “Collaboration may range from 
asynchronous, where an interactive activity is separated by long periods of time (e.g., e-mail, discussion 
groups), to synchronous, where an interactive activity is simultaneous (e.g., video conferencing)” (Lafifi, & 
Touil, 2010, p. 113-114). (3) Another change is paradigm shifting towards a positive direction regarding 
distance education facilitation and time commitment (Chronicle of Higher Education). The attitude and 
tactics of the instructor will affect the attitude of the learners. For example when an instructor talks so fast 
that students do not have time to think through a question before a facilitator goes to the next question, and 
neither party cannot see each other. (4) Further, understanding how to utilize the many technologies in order 
to multitask during class is important (Chronicle of Higher Education, 2010). The ability to handle several 
things at once leads to a smoother delivery, one without disturbances. (5) Finally is the need to adjust the 
amount of time required to prepare and then teach via distance education. Facilitators need to be cognizant 
of the direct impacts of on line education to positive learning. Moreover, learners will need to be able to 
apply the lessons to their specific understandings. 
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Facilitating in the distance learning arena is here to stay. To remain competitive and effective faculty will 
need to understand how to facilitate in the on line arena as well as teach face-to-face. Equally, administrators 
need to understand teaching through the distance education. They will need to view faculty development in 
several ways. First, for new instructors, administrators will need to develop a knowledge, skills, abilities, and 
competencies list. Next administrators need to establish on line teaching measurements for existing faculty. 
As faculty need to remain educated to the changes, so must administrators to meet the educational demands 
of the 2F‘ century. 

2.2 Understanding Technologies for Facilitating Online 

Understanding technology tools is important when researching and establishing an online faculty 
development program. Information Technology (IT) is met by facilitators’ slow acceptance of education 
technology. According to Duggan this “creates individualistic silos of technology” (Duggan, 2010, p. 3). 
“Effective incorporation and use of the technologies outlined in this paper will create an engaging 
environment that promotes joint learning in online courses” (Singh, Managalaras, & Taneja, 2006). 
Administrators and facilitators need to understand that simply housing technological tools and stating they 
are in-house is not the answer to promote success for distance learning professionals. Professionals need to 
comprehend the general use of educational tools, and then pin point specific tools for use. Additionally, they 
should understand tools for communication usage and tools for educational use. Attempting to know all of 
the education tools can engulf users; therefore; an educational development plan must be in place. Tools are 
delineated as educational and/or communication categories in table 1 . 


Table 1. Communication and Educational Technologies 


Category 

Example 

Explanation 

Educational 

Interactive White Boards 

Connects to a computer; A projector projects 
information on the board. Information is 
controlled by the finger, a pen, a stylus, or the 
like. 

Educational/Communication 

Web Conferencing - 
Audio/PowerPoint 

Tools that allow conferencing in remote places, 
through computers, tablets and smart 
phones/blackberries. 

Educational/Communication 

Web Training - Screen Sharing 

Tools that allow screen sharing of information. 

Educational/Communication 

Video Technology 

Tools used to move images for the purpose of 
viewing information. 

Educational/Communication 

Mobile Technologies 

Tools used for cellular technologies (auditory 
and visual capabilities). 

Educational/Communication 

Blogs 

Websites updated frequently with information 
in chronological order. 

Educational/Communication 

Wikis 

Sites created as collaborative efforts to share 
information; Can link from other sites. 

Educational/Communication 

Community of Practice 

Sites created as collaborative efforts for a 
group of people who share a common 
profession; Can link from other sites. 


Some instructors may have a few favorite tools that make their job of connecting with learners easier; 
however, this is not enough. To develop a competency in educational tools, instructors and administrators 
should understand the different categories and then be able to provide examples and explanations of them. 
This education leads to a process of continuous learning because new tools are consistently appearing. As 
well, older tools are gaining more distinction by adding new features. An example of this was the combining 
of a learning management system (used to deliver learning content) with a content management system (use 
to house content) to become the learning content management system (can develop, house, and deliver 
learning content) (Osuagwu, 2010). 

Another manner to understand and review tool is given by Singh, Managalaras, & Taneja, 2010. As part 
of a continuous learning plan, educational professionals can enhance their knowledge base of educational 
technology tools by targeting one to two tool categories per month to review and better understand. 
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Most professional are familiar with learning management systems such as Blackboard. Other potential 
technologies to gain a better understanding are noted in table two (2). 

Table 2. On line teaching tools classification . Adapted from Singh, Managalaras, & Taneja (2010). 


Ciitogor) 

Fimctionalitirs 

Examples 

Staging tools 

Pros ides the basic stnictiire to manage and 
deliver courses online. 

Learning m:uiageiiK'nt systems/course manageiiK'iit 
systems, (xilitw dix'unx'nt ntanageiiK'nt systeitis. 

Gxirse dcliven tixrls 

Enables the dissemination of course 
content in variixis forms. 

Onliiw videos, prxlcasts. news feeds, aixl screen 
capture. 

fmirse collaboration 
tix)ls 

.Alliws cxillaboration between stiKknts, 
gitHips. and the instructiir. 

Blogs, wikis, collaborative dix'ument management 
systenrs. 

Interactise 
communication t<x>ls 

Facilitates rich aunmunication between 
stmlents and instructors. 

Web cixiferencing. web b;ised simulations. 

.•XsscsMiKMlt and 
Ictirning tixds 

Assists in gauging student learning against 
\ari(«is objecti'es and benchmarks. 

Testing tixtls. cheating prevention tixrls. plagitnism 
detection tvxvis. 


2.3 Practical Applications of Educational Technologies in Distance Education 

Faculty need to understand practical experiences of applying different technologies and their applicability to 
fulfill different educational purposes. So far we have reviewed technological tools from the stance as 
educational and communication. Also, the tools have been viewed from the standpoint of categories, their 
functionalities, and with examples. Table 3 drills down through multiple distance education modalities while 
providing examples of the modalities, as well as the strengths and weaknesses. These tools can be used to 
teach courses, enhance course instruction, enhance instructor-learner and learner-learner interaction, and 
strengthen learners’ knowledge, skills, abilities, and competencies. To better understand and be able to apply 
these modalities instructors can target learning and practicing one or two per month. 


Table3. Distance education modalities: Examples, strengths, and weaknesses (Burton, Bessette, 2013) 


Modalities 

Examples 

Strengths 

Weaknesses 

Tele- 

Audio-graphics 

-Allows the interaction of 

All systems do not have the 

conferencing 

-GoToMeeting 

multiple users 

capability to: 


-Elluminate 

-Synchronous communication 

-Have participants raise hands. 


-Live Meeting 

channel 

automatically tally the total number 


-Netmeeting 

-User can share and use 

of raised hands, or flag the 


-WebX 

everything on their computers 
-Can be inexpensive to use if 
the medium of connection is the 
Internet 

-User can talk with a single 
person or large audiences 
-Participants can attend from 
remote locations 

instructors 

-Show thumbs down of have 
participants clap 

- Deliver good voice quality and 
without delays 

- Easily share the desktop, 
microphone and whiteboard 
-Have voice over IP (the Internet) 
-Display whiteboards 

-Modify the layout 
-Connect easily (Are plug-in 
required; if so can they be installed 
easily) 


Audio-conferencing 

-Allow students and teachers to 
replay sessions after the original 

-May have a different number of 
maximum participants 


-Elluminate Vroom 

end 

-May required a paid version for 


-ElashMeeting 

-Software uses voice over IP to 

conference calls 


-Google Talk 

allow user to communicate 

-May not be able to record sessions 


-Skype 

online 

-Select between push to talk or 
two-way capabilities 

-Different levels of communication 


28 


lADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2013) 


Modalities 

Examples 

Strengths 

Weaknesses 

Email 

-Microsoft 

-Google 

-Yahoo 

-Inexpensive manner to 
communicate with instructors 
and students 

-Can be used to instruct distance 
learning course 
- Access is 24/7 (anytime) 
-Unlimited licensing for users 

-Asynchronous communication 
channel 

-Internet connectivity is required 
and understanding the email 
program 

-Limited ability to display 
information (e.g., charts, graphs) 

Blogs 

-Teacher Personnel 
blog 

-Class blog 
-Student blogs 
-Education Award 
blog 

-Enabled learning to be 
independent of time and place 
-Learner-centered 
-Improves writing competencies 
-Captures paradigm shifts 

-Asynchronous communication 
channel 

-Content must remain fresh 
-Biased or inaccurate information 
can be interjected in the 
conversation 

Internet 

Web browsers 

-Explorer 

-Google Chrome 

-Mozilla 

Providers 

-Comcast 

-Sprint 

-Verizon 

-Allows a global audience, 24/7 
-Inexpensive 

-Immediate distribution of 
information 

-Create online discussion forums 
-Content publishing tool 

-Academic dishonesty 
-Can create a disconnect in conflict 
resolution as one can easily log-off 
-Possibility of getting inaccurate 
information 

-Hacking of personal information 
-Access to computer viruses 

Instructional TV 
(ITV) 

Educational 

Television 

-Public Broadcasting 

Service 

-Annenherg 

Eoundation 

Specialty Channels 

-Documentary 

channels 

-Public Affairs 

-People are generally 
comfortable with this medium 
-Can be inspiring and engaging 
-Delivery the has capability for 
high quality audio and video 
quality 

-Lessons can be recorded for 
future usage (flexibility) 

-Ease of access 

-Lessons are expensive to develop, 
and update 

-Required expensive video 
production time 
-The Instructor has to be highly 
concerned with appearance and 
expressions 

-Expensive for small audiences 

Learning Content 

Management 

Systems (LCMS)/ 

Content 

Management 

System 

-Blackboard 

-Generation21 

Enterprise 

-Saba 

TotalLCMS 

-Vuepoint 

-Enables the making or 
capturing, management, and 
transfer of content designed for 
education and/or performance 
support 

-Data is saved under catalog and 
tagged with metadata to permit 
search and recovery 

-May not be able to be operable with 
certain LCMS 

-May not support importing legacy 
content with minimal clean-up 

Learning 
Management 
Systems (LMS) 

-Dokeos 

-Jenzabar 

-Moodle 

-SPS 

-Manages learners (courses, 
completion rates, progress, 
scheduling, use notification, 
waitlists, etc.) 

-Enhances and supports 
classroom teaching and 
offerings, and allows self- 
registering for instructor-led and 
prepared training for educational 
institutions 

-Can be configured to either include 
of exclude features 
Instructors/Trainers may use the 
LMS and not consider other learning 
opportunities 

-All instructors cannot reproduce the 
sane face-to-face charisma through 
online delivery 
-Physical skills are not easily 
learned or tested online 

Mobile 

-Blackberry 

-Smartphones 

-Access to industry experts 
-Eollow conferences and 
seminars online 
-Learn through mobile 
applications for those with 
learning disabilities 
-Provides just-in-time 
performance support 

-Equipment costs 
-Asynchronous communication 
channel 

-Upgrades due to technology 
changing quickly 
-Battery life and recharging the 
phone 

-Storage space and costs 
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Modalities 

Examples 

Strengths 

Weaknesses 

Online Chat 
rooms 

-Chatroomss.com 
-Educational 
Technology Services 
-Education World 
-Gogloom 
-Moderated Chat 
Module 
-123FlashChat 
-PalTalk 

-Provide for two-way interactive 
exchange of information via the 
Internet; communicate in real 
time 

-Provide online office hours 
without the use of a telephone 
-Immediate response 
-Avenue to improve student’s 
thinking and literacy skills 

-May not have a moderated chat 
module where 

-May not have a whiteboard module 
-Participants need to schedule to 
meet 

- Can be addictive 

-Can access a computer virus 

Online Forums 
(e.g.. Bulletin 
Boards) 

-ProBoards 

-Popplet 

-Linoit 

-Provide for asynchronous 
communication amongst 
numerous participants 
- Forums can act as support 
systems for other teaching 
modalities; Can stand alone as a 
class 

-Delivers capabilities for 
participants read and re-read 
comments, and to think deeply 
and then respond to posts and 
comments 

-May be limited to the number of 
typed words 

-Threading may not be easy to 
follow 

-User are generally required to 
register 

Self-paced 

Learning 

-Audio recordings 
-CD Rom 
-MOOCs 
-Online tutorials 
-Text 

-Video recordings 
-On-the-job training 
aids 

-Supports anytime access (busy 
schedules) 

-Allows self-paced work for the 
highly motivated 
-Provides for case studies, 
motivating written discussions 
for learners to participate 

-Allows self-paced work which can 
be a drawback if the individual 
procrastinates 

-Instructors are challenged to think 
in advance regarding possible 
learner challenges 

Smartboards 

Interactive 

whiteboards 

-Enables collaboration on the 
Internet 

- Can share symbols and 
spreadsheets 
-Interacts through touch 

- Remote sites user may not be able 
to share information (depends upon 
the tool) 

-Will need audio-conferencing 
(telephone r internet) 

Video 

Interactive 

-Can work with learners at 
multiple locations at one time 
-Learners can view each other as 
each one works 
- Small set-up will may not 
require camera operators 

-Usage gets more difficult when 
multiple sites are in play 
- Active blackboard usage may be 
difficult to capture on screen 
-Large set-up will require camera 
operators 

Video 

Video Streaming 

-Live Broadcast 

-Video or audio content that 
does not have to be downloaded 
to play- content plays as it is 
received. 

-High cost due to the cost of the 
equipment needed 

Video 

Teleconferencing 

-Permits live contact, instructor 
to student, and student to student 
- Allows usage of boards, hand- 
outs (traditional classroom 
media), as well as recordings 
(audio and video), plus 
PowerPoint, and Overheads 
-Permits the introduction of 
guest speakers and listeners 
-Supports remote learners 

-Synchronous communication 
channel 

-Required the usage of cameras 
-May require large costly systems 
- Requires the understanding of 
what types of systems to use (may 
need an administrator) 

Video-Taping 

100% Video tape 

-May have poor video quality 

-Communication lost between 
instructor and student 

Wiki Space 

Online content 
storage space 

-Use as a knowledge repository 
-Store information in 
communities of practice 

-Needs Internet connectivity 
-Requires knowledge management 
and organization skills 
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3. CONCLUSION 

As distance education and exploding technological progressions reform the way instructors, administrators, 
and students conduct educational actions; general knowledge must keep pace with evolutions; therefore 
improved performance (Bessette & Burton, 2013). Tables one and two provide general knowledge about 
educational and communication tools. However, colleges, universities, and training departments need to use 
faculty education and developed programs which further explain these types of tools. Not only must the 
programs explain the tools and allow users to practice; they should provide individual development plans, 
targeted development (Burton et al, 2013). Such programs will enable users to become more knowledgeable 
and have the ability to use technological tools to better develop lessons. 

Education will need to be delivered through the adult learning theory, andragogy. This model of adult 
learning explains how adults learn. Andragogy is founded upon five postulations: (a) self-directedness, (b) 
need to know, (c) use of experience in learning, (d) readiness to learn, (e) orientation to learning, and (f) 
internal motivation (Knowles, 1984). 

Once better educated, faculty and administrators will be able to employ these technologies to develop 
lessons. At this point faculty will bring students into the fold of technology usage. The fold may be to 
become better with current usage, or to learn to use a new tool. Faculty development programs, drilling down 
to course development using tools, and then facilitating tool usage with students will help technology 
advancement in the educational arena. The education will be pushed through adult learning. 

This paper reviews educational and communication tool usages that can be used in a best practices 
program. On the other hand more study is needed to review research describing how these points offer 
benefits targeted development programs. 
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ABSTRACT 

This paper examines possible improvement to problem solving skills as a function of playing the video game Portal 2. 
Stealth assessment is used in the game to evaluate students’ problem solving abilities— specifically basic and flexible rule 
application. The stealth assessment measures will be validated against commonly accepted external measures. Eor 
instance, basic rule application will be correlated with Raven’s Progressive Matrices, and flexible rule application will be 
correlated with the Uses test, insight problems, and the compound remote association test. Improved problem solving 
outcomes will support our claim that Portal 2 can be an effective method to enhance problem solving skills— one of the 
most important cognitive skills in the 2U‘ century. 


KEYWORDS 

Problem solving; game-based learning; stealth assessment 


1. INTRODUCTION 

This paper describes our current research investigating the use of the video game Portal 2 (Valve 
Corporation) as a vehicle to assess and ultimately support problem solving skills in high school students. The 
reason this research is important is because in today’s interconnected world, being able to solve complex 
problems is, and will continue to be of great importance. However, students today are not receiving adequate 
practice solving such problems. Instead, they are exposed to problems that tend to be sterile and flat in 
classrooms and experimental settings (e.g., math word problems. Tower of Hanoi). Also, learning and 
succeeding in a complex and dynamic world is not easily or optimally measured by traditional types of 
assessment (e.g., multiple-choice responses, self-report surveys). Instead, we need to re-think assessment, 
identifying skills relevant for the 21st century — such as complex problem solving — and then figuring out 
how best to assess students’ acquisition of the skills. 

The organization of our short paper is as follows. We begin with a brief review of problem solving skills. 
Next, we discuss assessment in games focusing on a systematic approach for designing valid performance- 
based assessments (i.e., evidence-centered design). This is followed by our study design and outcome 
measures. We conclude with ideas for future research in the area. 

1.1 Literature Review 

1.1.1 Problem Solving Ability 

Problem solving has been studied extensively by researchers for decades (e.g., Gagne 1959; Jonassen 2003; 
Newell & Shaw 1958). It is generally defined as “any goal-directed sequence of cognitive operations” 
(Anderson 1980, p. 257) and is regarded as one of the most important cognitive skills in any profession as 
well as in everyday life (Jonassen 2003). There are several characteristics of problem solving as identified by 
Mayer and Wittrock (1996): (a) it is a cognitive process; (b) it is goal directed; and (c) the complexity (and 
hence difficulty) of the problem depends on one’s current knowledge and skills. 
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Researchers have long argued that a central point of education should be to teach people to become better 
problem solvers (Anderson 1980). And the development of problem-solving ability has often been regarded 
as a primary goal of the education process (Ruscio & Amabile 1999). But there is a gap between problems in 
formal education versus those that exist in real life. Jonassen (2000) noted that the problems students 
encounter in school are mostly well-defined, which contrasts with real-world problems that tend to be messy, 
with multiple solutions possible. Moreover, many problem-solving strategies that are taught in school entail a 
“cookbook” type of memorization, resulting in functional fixedness which can obstruct students’ ability to 
solve problems for which they have not been specifically trained. Additionally, this pedagogy also stunts 
students’ epistemological development, preventing them from developing their own knowledge-seeking 
skills (Jonassen et al. 2004). This is where good digital games (e.g.. Portal 2) come in— which have a set of 
goals and unknowns requiring the learner to generate new knowledge. 

Recent research suggests that problem solving skills involve three facets: rule identification, rule 
knowledge, and rule application (Westenberg et al. 2012; Schweizer et al. 2013). Rules in problem solving 
refer to the pattern of a problem and criteria of how to reach the goal. Because we will be using “stealth 
assessment” in the game (i.e., an unobtrusive and ubiquitous assessment embedded deeply within the game, 
e.g., Shute 2011), we will not be able to directly collect data on students’ rule identification and rule 
knowledge. However, since rule application is the outward expression of one’s rule identification and rule 
knowledge, the measurement of rule application will reflect students’ ability to identify rules and their rule 
knowledge. 

Any given problem in Portal 2 requires the application of either basic rules or rules that require cognitive 
flexibility— i.e., the ability to adjust prior thoughts or beliefs in response to a change in goals (Miyake et al. 
2000). Cognitive flexibility is the opposite of functional fixedness, defined as the difficulty that a person 
experiences when attempting to think about and use objects (or strategies) in unconventional ways (Duncker 
1945). Such cognitive rigidity causes people to view a particular type of problem as having one specific kind 
of solution without allowing for alternative strategies and explanations (Anderson 1983). Research has shown 
improved cognitive flexibility after playing first-person shooter (FPS) games (Colzato et al. 2010). The 
authors suggested that video games could be used to train older people to compensate for the decline in 
cognitive functions such as the ability to adapt to changes in the environment. However, empirical research 
examining the effects of video games on cognitive flexibility is still sparse. Our research is intended to begin 
to fill this gap. Below is the internal structure of problem solving skills that guided our research (discussed 
more in the next section on assessment): 



Figure 1. Internal Structure of Problem Solving Skill 
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1.1.2 Assessment with Evidence-Centered Design 

Assessments can be deficient or invalid if the tasks or problems are not engaging, meaningful, or 
contextualized. This calls for more authentic and engaging assessments, which has motivated our recent 
research efforts in relation to weaving assessments directly and invisibly within good games. In contrast, the 
amount of engagement in traditional (e.g., paper and pencil, multiple-choice) assessment is negligible. 
Another downside of traditional assessments is that they often invoke test anxiety, which can be a major 
source of construct-irrelevant variance. When these problems associated with traditional assessment — 
inauthentic and decontextualized items, and provoking anxiety — are removed (e.g., by using a game as the 
assessment vehicle), then the assessment should be more engaging. Additionally, if the assessment is 
designed properly, such as by using an evidence-centered design approach (Mislevy et al. 2003) then it 
should also be as, or even more valid than a traditional assessment. 

Evidence-centered design provides a way to assess students’ levels on target competencies by analyzing 
evidence extracted from the students’ interactions with the game or other complex learning environments 
(Shute 2011; Shute et al. 2009). Evidence-centered design includes three main models that work together: (a) 
competency model — which outlines the knowledge, skills or other attributes that are to be assessed; (b) 
evidence model — which establishes the statistical links between the competencies and their associated 
metrics; and (c) task model — which identifies the features of tasks that can elicit the necessary evidence. 
Examples of indicators of the competency in the game will be provided in the external outcome measures 
section. 

Eor example, in Portal 2, if a player follows basic rules such as avoiding harmful objects (e.g., turrets and 
acid river), or making use of the tools and other objects in the environment (e.g., refraction cubes and light 
bridges), this provides evidence that the player is competent at basic rule application. Particular facets will be 
modeled by different game levels selected by us with variation in difficulty. The students’ competency will 
be measured by the time it takes them to finish the tasks. 


2. METHOD 

2.1 Participants 

Around 220 11-12* graders from a high school located in the northern Elorida will be recruited for the study. 
We will try to have equal number of girls and boys to control for any gender effects related to gaming. 
Demographic data will be collected from all participants. Students who are expert gamers or have beat Portal 
2 before will be excluded from the study. Half of the students will be randomly assigned to the control group, 
which will play a web-based “brain trainer” game called Lumosity (which claims to improve problem solving 
skills). The other half will be assigned to our experimental condition, playing Portal 2 (which is a 
commercial game making no claims relative to learning). Students will be compensated with $100 for full 
participation (i.e., 14 hours of gameplay and 2-3 hours of pretests and posttests— our external measures). 

2.2 Design 

Students will play their assigned game for 14 hours. The entire study will be completed at the students’ home 
at their own pace— within a 3-week window. After game play for both groups, students will be instructed on 
how to extract the gameplay log files from their computer and upload it for our analysis. Students will also 
take online-based problem solving tests both before and after the intervention. This is intended to (a) validate 
our stealth assessment measures of problem solving skill, and (b) provide data relative to learning from the 
game. We hypothesize that students’ problem solving skills will improve after the game intervention, 
possibly more in the experimental than control condition despite the control condition’s game being 
explicitly touted as support for such skills. 
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2.3 Materials 

Portal 2 is a first-person, puzzle-platform, “shooter" game, consisting of a series of provocative puzzles. 
These puzzles must be solved by teleporting your player’s character and various objects using the “portal 
gun,” a device that can create inter-spatial portals between two flat planes. To solve the progressively more 
difficult challenges, players must figure out how to locate, obtain, and then combine various objects 
effectively to open doors and navigate through the environment, with dangers abounding. Portal 2 provides a 
unique environment that can promote problem solving skills through providing players extensive practice 
figuring out solutions to complex problems on their own. We posit that problem solving skills learned in the 
game can be transferred beyond the immediate game environment. 

Lumosity, the intervention for the control group, is a web-based platform that hosts several small-scale 
games. Advertisements for Lumosity claim the games were designed by neuroscientists, and improve brain 
health and performance. Some of the games especially feature problem solving and cognitive flexibility. 
They also claim that the games provide personalized training to different users and that 10 hours of Lumosity 
training creates drastic improvements. 

2.4 Assessment in Portal 2 

Stealth assessment of problem solving will be embedded in the game. We will use original levels in the game 
as well as several customized levels from the community that help elicit specific evidence for our 
competency — problem solving ability. A total of 75 levels were collected for the research. Basic and flexible 
rule application load on different levels with varying weights. A level may be easy on basic rule application 
but difficult on flexible rule application. Below are examples of how we designed/selected levels to elicit 
evidence for the two facets of rule application: 

• Basic rule application: 

Basic rules in Portal 2 are rules directly instructed or can be picked up easily. For example, players 
should be able to learn that the river is hazardous from the cueing picture on the floor near the river. Or if a 
player fails to notice it and falls into the river, he will die and resurrect from the last automatic saving point. 
Afterwards, he should be aware of the rule. Other basic rules relate to avoiding laser beams, knocking over 
turrets to terminate them, putting a cube on the weighted button to activate any device connected to it, etc. 
Task modeling of basic rule application involves the manipulation of the number of rules present in a level 
and the difficulty of the rules to be included. 

• Flexible rule application: 

Flexible rules in Portal 2 refer to rules that can only be inferred from the basic rules. For example, the 
basic rule is that the weighted button can be activated by the weight of a cube. Cognitive flexibility requires 
players to realize that the body weight of the player may be a replacement when a cube is not readily 
available. Other flexible rules in the game include the use of the hard light bridge to catch a falling cube or to 
hold it above a destination (e.g., a weighted button to be pressed) and release it after a sequence of actions to 
be performed before the release. Similar to the task modeling of basic rules, the modeling of flexible rule 
application involves the number of rules present in a level, the difficulty of a single rule, and the combined 
use of different rules. 

Log files that record students’ performance in gameplay are extracted by enabling the developer console 
of the game. Students’ problem solving performance can be assessed by information in the log files such as 
the time it takes to solve a level, number of steps to reach the exit door of each level, the total number of 
solved puzzles, and the number of attempts before successfully complete specific steps. 

Examples of specific indicators per facet of problem solving from the game can be seen in the following 
figure: 
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Eigure 2. Competency Model of Problem Solving Skills with Indicators from Portal 2 

2.5 External Outcome Measures 


The stealth assessment of students’ problem solving skills will be validated against external measures of 
problem solving. Two sub-facets of rule application (i.e., basic rule application and cognitive flexibility) will 
be measured. Basic rule application will be measured by Raven’s Standard Progressive Matrices. The test 
requires participants to infer the pattern of the missing piece from the other patterns given. Although the test 
is widely used as an intelligence test (e.g., Prince et al. 1996; Rushton & Jensen 2005), as Raven (2000) 
pointed out, the Raven’s Progressive Matrices focus on two components of general cognitive ability-eductive 
and reproductive ability. Eductive ability involves making meaning out of confusion and generating 
high-level schema to handle complexity. Reproductive ability is the ability to recall and reproduce 
information. In Portal 2, for example, players are instructed that the laser beam is deadly. If the player knows 
this rule, she should realize that the turret is also harmful since it emits a laser beam. We have selected 12 
items from the Raven’s Progressive Matrices test for the pretest and 12 items for the posttest. 

Cognitive flexibility will be measured by three tests: alternative uses, insight problems, and remote 
association. 

The Alternative Uses test, developed by Guilford in 1967, requires respondents to find unusual uses for 
commonly seen objects. For example, the common use for a newspaper is to read it. The test requires 
examinees to “think of other uses the object or part of the object could serve.” Answers may be something 
like to start a fire, to wrap objects, or to make up a kidnap note (Wilson et al. 1953). We have created four 
items in the pretest and four items in the posttest and students are allowed four minutes to work on each item. 

We posit that playing Portal 2 can improve performance on the test because many tasks in the game 
require the players to come up with different uses of a given tool or to deal with a problem via different 
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methods. For example, the original function of the hard light bridge is to serve as a bridge for the player to 
walk on. Later, the players need to come up with new uses of the hard light bridge, such as serving as 
barricade to block turrets. 

Insight problems are intended to yield an “Aha” moment for problem solvers when the solution occurs 
after a short or long moment of bafflement (Chu & MacGregor, 2011). Insight problems require problem 
solvers to shift their perspective and look at obscure features of the available resources or to think of different 
ways to make use of an object. We have selected five insight problems for the pretest and five for the 
posttest. For instance: Marsha and Marjorie were born on the same day of the same month of the same year 
to the same mother and the same father yet they are not twins. How is that possible! The answer is that they 
are triplets. The question is not particularly hard, but it requires problem solvers to break from routine 
thinking and think beyond the immediate context. The posttest will be an alternative form of the pretest and 
reliability of the tests will be examined during the study. 

The Remote Association test was originally developed by Mednick (1962) to test creative thought without 
any demand on prior knowledge. Each item consists of three words and problem solvers are required to find 
the solution word associated with all words that appear to be unrelated. The fourth word can be associated 
with each of the three words in multiple forms, such as synonymy, formation of a compound word, or 
semantic association (Chermahini, Hickendorff, & Hommel, 2012). For example, the answer to the triad 
night/wrist/stop is “watch.” Schooler and Melcher (1995) reported that problem solvers’ success on this test 
correlates with their success on classic insight problems. We have selected five items for the pretest and five 
for the posttest. 


3. DISCUSSION AND IMPLICATIONS 

We are currently recruiting participants for this study which is scheduled to run during the summer of 2013. 
In addition to the full set of Portal 2 levels, we have collected levels from the community and also created 
additional levels (with the Portal 2 “modding” tool) that will allow us to array levels by difficulty to provide 
for adaptive challenges to support the “zone of proximal development” (Vygotsky, 1987) and “flow” state 
(Csikszentmihalyi, 1990). By the time of the CELDA 2013 conference, we will have collected all of the data 
and we will present our findings. 

As mentioned earlier, problem solving is one of the most important cognitive skills in any profession and 
in everyday life (Jonassen, 2003). Positive findings from this study will support the effectiveness of video 
games in developing this important skill. Using games as assessment vehicles sounds like a great idea, but 
there are some issues that need to be addressed. Eor instance, there are potential sources of error variance in 
videogame assessments such as the level of interest in the target game. However, we believe this will not be a 
problem with Portal 2 given its broad appeal (e.g., over 3 million copies have been sold since it came out last 
year, according to GamePront). In short, we believe that Portal 2 can be used to assess problem solving 
better than traditional assessments by virtue of having more authentic, contextualized, and very engaging 
tasks. 
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ABSTRACT 

The objective of this study was to determine if the time spent on Facebook and the purpose for which Facebook was used 
had any impact on the academic achievement of the students. This exploratory research used a questionnaire to collect 
data from 25 1 undergraduate students at a private university in Bangkok, Thailand. Data were analyzed using Multiple 
Regression Analysis to the conclusion that students used Facebook on an average of one hour and thirty minutes, for the 
purposes of communication and collaboration; that the number of hours spent on Facebook had no effect on their 
academic achievement; that using Facebook for communication impaired academic achievement; and, that using 
Facebook for collaboration did not impair academic achievement. The study has also found that time management is a 
key predictor in the determination of the students’ academic achievement. 


KEYWORDS 

Facebook, academic achievement, time management 


1. INTRODUCTION 

Social networks are defined as a body of applications that augment group interaction and shared spaces for 
collaboration, social connections, and aggregates information exchanges in a web -based environment 
(Barlett-Brag, 2006). Facebook is one of the most commonly known social networks, particularly among the 
young individuals who are still studying. There have been reports that as many as 85 to 99 percent of all 
students are on Facebook (Hargittai, 2008; Jones & Fox, 2009; Matneycfe Borland, 2009). Thailand is reported 
to have as many as fourteen million Facebook users -the third largest country of Facebook users in Asia 
(Go-Gulf.com, 2012). Facebook provides an opportunity to users, to create personalized profiles that include 
general information like education background, work background, and favorite interests and also to add links 
and song clips of their favorite bands, post messages on friends' pages, and post and tag pictures and videos, 
among other things (Rosmarin, 2007; Zywica & Danowski, 2008). 

As many students are using Facebook daily, Facebook seems to offer great potentials for teaching and 
learning. Students take advantage of Facebook as a tool in their studies as this platform makes it possible for 
them to exchange ideas and opinions or to interact with the other users at multiple levels. Several educational 
institutions have integrated Facebook into their management, for instance, to promote the institution, to 
communicate with their personnel, and to communicate with the students. Teachers have used Facebook as 
an instructional tool inside and outside of classrooms as well to bring an air of freshness to the learning 
environment. Facebook is consistent with the requirements of the learners and makes it possible for the 
teachers to connect with the learners more quickly. It allows for better participation in the learning activities 
as well. 

Facebook is like a stage that gives learners more courage to voice their opinions than in classrooms 
(Cheung, Chiu & Lee, 2011; Pempek, Yermola & Calvert, 2009; Roblyer, McDaniel, Webb, Herman & 
Witty, 2010). However, with the widespread popularity of Facebook among the students, with more users 
online and with more hours spent on Facebook, a question arises about how Facebook affects the students’ 
academic achievement. Facebook can help improve the performance of the students while at the same time it 
may hinder their learning process as well. 
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This is because the use of Facebook in classroom can distract students. With extended use in combination 
with poor time management, the students may have less time for their studies and assignments, resulting in 
procrastination and eventually poorer academic achievement. It is, therefore, interesting to study if Facebook 
affects the academic achievement of students. 

1.1 Theoretical Background 

Cognitive Load Theory (CLT) basically states that a learner’s attention and working memory is limited. This 
limited amount of attention can be directed towards intrinsic, germane, or extraneous processing. Intrinsic 
processing describes a learner’s focus on the learning content and its key features; it is determined by the 
intellectual demands of learning content (the complexity of the content). Germane processing describes a 
deeper processing of the content by its organization to cognitive representations and its integration into 
existing representations (integrating previous knowledge). Finally, extraneous processing describes cognitive 
demands during learning, which do not foster the actual objectives of the learning material, for example 
cross-references or navigation elements (Swello 1988). 

1.2 Time Management and Academic Achievement 

Outside of the classroom, if students cannot manage their time effectively since they spend too much time on 
Facebook for pleasure, they do not take the time to read the assigned material. As a result, they usually 
request for extensions or have late submissions. Moreover, they are not well -prepared for examination 
resulting in failing because they have very out-of-class study time. To conclude, students with good time 
management tend to have good academic achievement. 

Trueman and Hartley (1996) studied the time management practices of American students and found 
female students to manage time more effectively than male students. Time management skills are crucial to 
academic performance. Good time management leads to better academic performance as well. Sansgiry, 
Kawatkar, Dutta and Bhosle (2004) discovered that time management skills, academic competence and 
learning strategies are the factors essential to effective learning. Griffin, MacKewn, Moser and VanVuren 
(2012) found that time management is positively related to the GPA of the students. Those with good time 
management achieve higher cumulative GPA. 

1.3 Facebook and Academic Achievement 

Several scholars have taken interest in the influence of Facebook on academic performance. The findings are 
not conclusive so far they are not in alignment: certain researches find Facebook to have caused poorer 
academic performance while other researches find Facebook to have no correlation with academic 
performance. The study of Kirschner and Karpinski (2010) surveyed the use of Facebook of undergraduate 
and graduate students to determine how it affected their academic performance. Taking into account the 
students’ GPA, Kirschner and Karpinski (2010) found the GPA of students using Facebook to be lower than 
those who were not on Facebook. The use of Facebook has left the students with less time for their studies. 
On the contrary, O’Brien (2011) did a dissertation about the use of Internet and Facebook by the students. In 
this study, the Facebook usage behaviors and perception of the impact of Facebook on academic performance 
were examined. The study revealed that students spent an average of two hours on Facebook and 80% of that 
time was spent on socializing and entertainment. It revealed no significant relationship between Facebook 
usage patterns and academic performance. This is consistent with the findings of a study conducted by Pasek, 
More and Hargittai (2009) which did not find a relationship between the use of Facebook and the GPA of the 
students. The objective of this research was to study how the time spent on Facebook and the purposes for 
which Facebook was used affected the student’s academic achievement, as determined from their cumulative 
GPA. The other factors of interest with influence on academic achievement were gender and time 
management skills. 
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2. METHOD 

2.1 Respondents 

The population in this study was undergraduate students at a private university in Bangkok, Thailand. A visit 
was made to the private university in July 2012 on a scheduled basis with prior permission from the 
instructors to enter their classes. The research tool used was an anonymous self-report questionnaire. 
Regarding ethical issues, the students were asked if they would be interested to volunteer by completing the 
survey. So, all respondents in this study were voluntary; they had been informed that they could stop 
responding to the items whenever they felt uncomfortable. Students spent approximately ten or fifteen 
minutes answering the questions. They were asked to sign a consent form that briefly described the study 
before taking part in the study. They were also told that the results from this study would not affect their 
course grades. There were 268 students completing the questionnaire. Of this number, 251 copies or 93.7% 
of the completed questionnaires were found to be usable. That was sufficient for further analysis. 

2.2 Instrument 

The instrument used to collect data was a questionnaire with three parts. The first part contained questions 
about their socio-demographic profile including gender, class year, cumulative grade point average, and time 
spent on Facebook in each day. The second part contained eleven questions about their usage of Facebook for 
academic purposes (e.g. “How significantly do you use Facebook to access course notes and other 
materials?”, “How extensively do you use Facebook to communicate with other students in your class?). The 
third part contained five questions about the students’ time management practices (e.g. “1 set a goal of what 1 
must get done this week.”). The responses were based on the Likert scale from one to five. 

The questionnaire was inspected for accuracy of content and for appropriateness of the questions by three 
qualified experts in the teaching field. Since the questionnaire was originally prepared in Thai, it was 
translated into English by a linguistic specialist from the Language Institute, Bangkok University. This 
questionnaire was piloted with fifty students and a reliability analysis was performed. The inter-item 
reliability (Cronbach’s alpha) of Facebook usage for academic purposes was 0.876, and 0.652 for time 
management. 

2.3 Data Analysis 

Descriptive statistics was used initially including frequency, mean, standard deviation for description of 
sample group demographics. The questions were grouped for Facebook usage for academic purposes using 
the Exploratory Eactor Analysis. Subsequently, an analysis was conducted to determine the variables 
predicting the cumulative GPA of the students using the Multivariate Regression Analysis. 


3. RESULTS 

3.1 Socio-Demographic Profiles of the Respondents 

The respondents in this study were 43.6% female and 56.4% male, aged between 18 and 21 years. The 
number of respondents from each class year was similar. The cumulative GPA of the sample students had a 
mean of 2.88 (SD = 1.07) with the lowest of 1.25 and the highest of 3.91. Students spent an average of 91 
minutes on Eacebook (SD = 74 minutes). The student who was on Eacebook the longest spent three hours on 
it each day while the student who spent the shortest time on Eacebook was on it for ten minutes. The details 
can be seen in Table 1. 
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Table 1. Demographic Information of the Respondents 



Frequency 

% 

Gender 

male 

144 

57.4 

female 

107 

42.6 

Class year 

freshman 

51 

20.3 

sophomore 

72 

28.7 

junior 

68 

27.1 

senior 

60 

23.9 

Cum. GPA 

0.00 - 2.00 

29 

11.6 

2.01-2.50 

83 

33.1 

2.51-3.00 

81 

32.3 

3.01-4.00 

58 

23.1 

Eacebook time 

Less than 30 min. 

80 

31.9 

31-60 min. 

43 

17.1 

61 - 120 min. 

55 

21.9 

more than 120 min. 

73 

29.1 


3.2 Factors Analysis Results 

The eleven statements concerning the use of Facebook for academic purposes by the students were analyzed 
using principle components analysis with varimax rotation method to determine the underlying dimensions. 
In this process, the minimum Eigen value of 1.0 was used as cut-off. Only the constituent statements with 
factor loadings of more than .5 were retained. The Kaiser -Meyer-Olkin (KMO) measure of sampling 
adequacy and the Bartlett's test of sphericity were used to test the fitness of the data. The result of KMO was 
0.852 and the Bartlett's test of sphericity was found at the significance level of 0.000. These figures 
suggested that the use of factor analysis was appropriate (Hair, Anderson, tatham, & Black, 1998). Two 
dimensions were extracted from the eleven statements. The factors derived were communication and 
collaboration. Communication means using Facebook to contact or communicate with classmates or 
instructors while collaboration refers to using Facebook to do assignments together in groups or to raise any 
academic issues to be discussed among friends. The cumulative percentage of explained variance was 61.23 
% which meant that the two factors could explain 61.23 % of variation of students’ usage of Facebook for 
academic purposes. The details can be seen in Table 2. The factor score of the two factors were used as 
predictors of students’ academic achievement in the next step. 
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Table 2. Result From Factor Analysis of Using Facebook for Academic Purposes 


Factors 

Factor 

Eigen- 

%of 


loading 

value 

variance 

Communication 


4.958 

45.08 

Communicate with other students in my course 

.827 



Participate in activities prepared by the instructors 

.812 



Participate in activities or games which are a part of the coursework 

.781 



Communicate with university staffs 

.746 



Communicate with the instructors 

.743 



Keep track of university announcements 

.603 



Access to course notes and other materials 

.509 



Collaboration 

Exchange information with classmates 

.845 

1.777 

16.15 

Send and receive information among friends with the aim to complete 

.822 



group reports 




Exchange views with friends on class subject matters 

.803 



Complete quizzes provided by the teachers. 

.598 



Total % of variance 



61.23 


3.3 Predictors of Students’ Academic Achievement 

For the analysis of variables predicting the students’ academic achievement, the Multivariate Regression 
Analysis was used. Academic achievement was determined by the cumulative GPA of the students. The five 
predictors were gender, time spent on Facebook, time management skills and the two variables identified by 
the Factors Analysis which were communication factor score and collaboration factor score. 

Gender values of “0” and “1” were assigned for female and male respectively. The variable of time spent 
on Facebook was converted to standard score in order to present it as normal distribution. The results of this 
analysis are presented in Table 3. 


Table 3. Regression with Cumulative GPA and the Five Predictors 


Predictors 

B 

SEB 

P 

t 

Constant 

-1.074 

0.426 


-2.521 

Gender 

-0.342 

0.127 

-0.169 

-2.690** 

Time management 

0.385 

0.122 

0.196 

3.145** 

Time spent on Facebook 

-0.045 

0.062 

-0.045 

0.469 

Facebook for communication 

-0.190 

0.061 

-0.190 

-3.088** 

Facebook for collaboration 

0.023 

0.062 

0.023 

0.362 

F= 6.138**, R^= 0.113 


Remarks 1) ** p-value< 0.01 

2) Time spent on FB was converted to standard score 
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A test at the significance level of 0.01 has found three predictors to be capable of predicting the 
cumulative GPA of the students: (1) gender (P = -0.169; Male students had lower cumulative GPA than 
female students.); (2) time management (P = 0.196; Time management had a positive influence on GPA. 
With good management skills, the students would have good academic performance as well, and (3) use of 
Facebook for communication (P = - 0.190; The use of Facebook for communication had a negative impact on 
GPA. Their academic performance was impaired if students used Facebook to communicate with their 
classmates too extensively. 

Upon the review of the magnitude of standardized regression coefficients (P), it is noticed that the P of 
time management is highest (P = 0.196), suggesting that time management is the best predictor of academic 
achievement. The factors of time spent on Facebook and the use of Facebook for collaboration did not predict 
students’ academic achievement. 


4. DISCUSSION AND CONCLUSION 

Thai students use Facebook on an average of one hour and thirty minutes per day. This is similar to the 
survey findings involving a sample of American students who use Facebook on an average of one hour and 
forty minutes per day (Junco, 2012). However, the average time of spending Facebook was found to be a bit 
different from the finding in a study revealing that students spent an average of two hours on Facebook and 
80% of that time was spent on socializing and entertainment (O’Brien, 201 1). 

Gender is one of the factors that should be brought to discuss. Female students have higher cumulative 
GPA than male students. The finding of this study is consistent with that of the studies by other scholars 
where it is found that females often outperform males in collegiate (i.e.Sheard, 2009; Barrow, et al., 2009; 
Farsides & Woodfield, 2007). 

The next discussion on factor is time management skills. It is worth noting that students with good time 
management achieve higher cumulative GPA. Since time management is found to be the best predictor of 
academic achievement in this study, it is necessary to warn students to be aware of time management. This 
result is consistent to previous studies (Sansgiry, Kawatkar, Dutta & Bhosle, 2004; MacKewn, Moser & 
VanVuren, 2012) in that time management skills are the factors essential to effective learning. Although time 
spent on Facebook is not a predictor of academic achievement in the present study, students should realize 
that spending too much time on Facebook has left the students with less time for their studies. 

Generally, the use of Facebook for academic purposes is for communication and collaboration. This study 
found that the use of Facebook for communication has a negative effect on academic achievement as students 
tend to discuss matters not related to their studies, resulting in a waste of time. On the other hand, this study 
indicates that the use of Facebook for collaboration does not have an influence on academic achievement. A 
reason for this finding is that Thai students are not accustomed to using Facebook for collaboration in doing 
assignments; academic contents do not much attract their attention and interest. This can be explained on the 
basis of Cognitive Load Theory stating that a person’s attention is limited. This limited amount of attention 
can be due to the complexity or difficulty of the content. Therefore, a focus on the content is not long if 
Facebook is used for academic purposes. Therefore, the effect on their academic achievement is not very 
evident. 
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ABSTRACT 

The objective of this research was to examine how Thai university students used Eacebook for academic purposes and 
how public university students differed from private university students in this regard. This was an exploratory research 
where a questionnaire was used to collect data from 460 undergraduate students in Bangkok, Thailand. The data were 
then analyzed using Multivariate Analysis of Variance. The study found that Thai students used Eacebook more to 
communicate and to share than to collaborate. This communication or sharing of information was more among the 
students than with their professors. It also found that private university students used Eacebook to collaborate more than 
their counterparts at public universities. 


KEYWORDS 

Eacebook, communication, sharing, collaboration 


1. INTRODUCTION 

Facebook is a significantly popular online social network. According to a survey of Go-Gulf.com conducted 
during the first three months of 2012, there are as many as 901 million Facebook users worldwide with more 
than 14 million of those being in Thailand -Asia’s third largest country where Facebook users number is 
concerned (Go-Gulf.com, 2012). Other surveys in various countries of the world have similarly confirmed 
that Facebook is the most popular online social network among university students. There have been reports 
that as many as 85 to 99 percent of all university students are on Facebook (Hargittai, 2008; Jones & Fox, 
2009; Matney & Borland, 2009). 

Facebook offers applications people can use to communicate, to exchange opinions and to interact at 
multiple levels. It is basically used as a platform for social interaction. Facebook users can be both content 
creators and readers. For these characteristics, several leading educational institutions have adopted Facebook 
as a means to further develop their management. Therefore, Facebook has been used by many institutions in 
their communication with personal and with students. Teachers can use Facebook as a supporting tool inside 
and outside of their class to give the appearance of being in keeping with the times. The use of Facebook as 
an instructional tool is consistent with the needs of the learners as well. It allows the teachers to communicate 
with their students more quickly, encourages participation and involvement, as well as serves as a venue 
where learners can more openly voice their opinions, compared to traditional lecture rooms (Cheung, Chiu, 
& Lee, 2011; Pempek, Yermola, & Calvert, 2009; Roblyer, McDaniel, Webb, Herman, & Witty, 2010). 

The study of Tiryakioglu and Erzurum (2011) surveying university teachers’ opinions about social 
networks particularly Facebook as an educational tool found that 63.3 % of them agreed that Facebook 
helped communication between teachers and their students more effectively. Meanwhile, 61.2 % stated that 
Facebook was a tool for students to acquire information from classroom or institution. 58.3 % agreed that 
Facebook supported in grouping students on a basis of their interests or needs while 54.2 % believed that it 
was the effective tool which greatly affected the learner group’s performance. 
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Furthermore, Grosseck, Bran and Tiru (2011) conducted a survey on the use of Facebook by university 
students in Romania. They found that most students used Facebook more for social purposes (e.g. keeping in 
touch with friends and family members, sharing photographs and tagging photographs) than for academic 
purposes. 

Nowadays, significant numbers of Thai university students are on Facebook as well. Nevertheless, the use 
of Facebook for academic purposes may or may not be successful as it is dependent upon several factors such 
as the readiness of the institution in the procurement of equipment for online instruction, the readiness of the 
teachers in the use of Facebook inside and outside of classrooms, and the reception and the needs of the 
students where this is concerned. 

Facebook are used by students mostly to discuss with their friends about assignments given by their 
professors or the lectures they have listened to. Students can use Facebook to exchange lecture notes and 
other materials. In their view, Facebook is a beneficial tool for education. They feel that Facebook is indeed a 
low-cost tool for the enhancement of knowledge in higher education. 

In Thailand, undergraduate curricula are offered at public and private universities. The two types of 
institutions differ in terms of environment, personnel and ICT infrastructure. The students from public and 
private universities are typically of different natures as well. Public university students are more focused with 
their studies than private university students. Generally, private university students do not attend the class, so 
instructors need to find some strategies to attract their attention. One of those strategies is to use Facebook as 
an instructional tool. It is a popular social networking site equipped with tools designed for social interaction 
that instructors can organize for academic uses. Thai instructors can use Facebook as a channel to 
communicate with students and as a platform to deliver instructional content. Therefore, it is interesting to 
explore how university students are using Facebook to collaborate on academic related tasks. 

The objective of this study is to compare the use of Facebook for academic purposes of public university 
students and private university students. Usage in this study is classified as communication, sharing and 
collaboration. 


2. METHOD 

2.1 Respondents 

The population of this study was undergraduate students in Bangkok, Thailand. A total of 486 students were 
randomly selected at both public and private universities. Research assistants assisted with the conduct of the 
survey during class hours by distributing anonymous self-report questionnaire to the students and asking 
them to complete it in approximately ten minutes. Upon reviewing the completed questionnaires, a total of 
460 copies or 94.7% of all were found to be valid. This was sufficient for further analysis. 

2.2 Instrument 

The tool used to collect data was a two-part questionnaire. The first part contained questions about their 
socio-demographic profile including gender, class-year and time spent on Facebook each day. The second 
part contained eleven questions about their usage of Facebook for academic purpose. The questions covered 
three aspects: communication, sharing and collaboration. The responses were evaluated on the basis of the 
Likert scale from one to five. This questionnaire was examined by three teachers for content validity and 
piloted with fifty students and a reliability analysis was performed. The inter-item reliability (Cronbach’s 
alpha) of these items was 0.737, 0.654 and 0.776 respectively. 

2.3 Data Analysis 

Descriptive statistics was used initially including frequency and percentage for description of sample group 
demographics and students’ use of Facebook for academic purposes. Then data of Facebook usage got from 
public university students and private university students were compared using mean, standard deviation, and 
Multivariate Analysis of Variance (MANOVA). 


48 



lADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2013) 


2.4 Results 

2.4.1 Socio-Demographic Profiles of the Respondents 

There were 460 respondents in this study. Overall, 53% were females and 47% were males, aged between 18 
and 21 years. Of all respondents, 47.6% were public university students and 52.4% were private university 
students. The samples of public universities had a bit more females than males while those of private 
universities had a bit more males than females. The number of samples got from private universities was 
similar in four levels of years ranging from freshman to senior, but the great number of those got from public 
university were sophomore. Private university students had a little more average time of using Facebook than 
public university students. That is, private university students had an average Facebook time of one hour and 
twenty-eight minutes while public university students had an average Facebook time of one hour and twenty- 
two minutes a day. The detail was shown in Table 1. 


Table 1. Socio-Demographic Profiles of the Respondents 



Public University (n=219) 

Private University (n=241) 


Frequency 

% 

Frequency 

% 

Gender 





male 

104 

47.5 

136 

56.4 

female 

115 

52.5 

105 

43.6 

Class year 





freshman 

49 

22.4 

49 

20.3 

sophomore 

80 

36.5 

63 

26.5 

junior 

48 

21.9 

68 

28.2 

senior 

42 

19.2 

61 

25.3 

Facebook time 

Mean = 82.19 min 


Mean = 88.24 min 



SD = 81.11 min 


SD = 69.30 min 



2.4.2 Usage of Facebook for Academic Purposes by Thai students 


From Table 2, frequency and percentage of students marking on each statement were shown in five rating 
scale. There were eleven statements about the use of Facebook for academic purposes which belonged to 
three aspects. Regarding the aspect of communication, students used Facebook to communicate with other 
students in their course the most. That is, 44.6 % of them rated strongly agreed and 42.8 % chose to agree 
with this statement. As for the sharing aspect, it was found that percentage of the students rating “strongly 
agreement and agreement” on the three statements did not differ. Concerning collaboration, accessing course 
notes and other materials were chosen by Thai students the most when compared with the other three 
statements. 
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Table 2. Frequency and Percentage of Using Facebook for Academic Purposes of Thai Students 


Facebook Usage for Academic Purposes 

Extremely 

Very 

Erequency 

Somewhat 

Not 

very 

Not at 
all 

Communication 


205 

197 

49 

8 

1 

1 . Communicate with other students in my course 

(44.6) 

(42.8) 

(10.7) 

(1.7) 

(0.2) 

2. Communicate with the instructors 

31 

(6.7) 

83 

(18.0) 

182 

(39.6) 

118 

(25.7) 

46 

(10.0) 

3. Communicate with university staffs 

11 

(2.4) 

56 

(12.2) 

141 

(30.7) 

145 

(31.5) 

107 

(23.3) 

4. Keep track of university announcements 

62 

(13.5) 

167 

(36.3) 

134 

(29.1) 

65 

(14.1) 

32 

(7.0) 

Sharing 

5. Exchange information with classmates 

156 

(33.9) 

204 

(44.3) 

81 

(17.6) 

12 

(2.6) 

7 

(1.5) 

6. Send and receive information among friends with the 
aim to complete group reports 

138 

(30) 

199 

(43.3) 

94 

(20.4) 

24 

(5.2) 

5 

(1.1) 

7. Exchange views with friends on class subject matters 

96 

(20.9) 

201 

(43.7) 

122 

(26.5) 

31 

(6.7) 

10 

(2.2) 

Collaboration 

8. Access course notes and other materials 

120 

(26.1) 

189 

(41.1) 

108 

(23.5) 

37 

(8.0) 

6 

(1.3) 

9. Participate in activities with the teachers 

35 

(7.6) 

75 

(16.3) 

192 

(41.7) 

111 

(24.1) 

47 

(10.2) 

10. Complete quizzes provided by the teachers 

38 

(8.3) 

98 

(21.3) 

182 

(39.6) 

95 

(20.7) 

47 

(10.2) 

1 1 . Participate in activities or games which are a part of 
the coursework 

37 

(8.0) 

83 

(18.0) 

186 

(40.4) 

98 

(21.3) 

56 

(12.2) 


2.4.3 Comparison of Facebook Usage between Public University Students and Private 
University Students 


Facebook usage for academic purposes were categorized into three aspects namely communication, sharing, 
and collaboration. Regarding communication, both public and private university students used Facebook to 
communicate with other students in their course the most, followed by keep track of university 
announcements. However, it was interesting to see that public university students had higher mean scores 
than private university students in these two statements. 

Table 3 also revealed that private university students used Facebook to collaborate more than their 
counterparts at public universities. That is, private university students used Facebook to complete group 
reports and exchange views with friends on class subject matters more than public university students while 
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exchanging information with classmates was used more by students in public universities than those at 
private universities. 

When the four statements in collaboration aspect were examined, it was found that private university 
students used Facebook to collaborate more than their counterparts at public universities. Although students 
in both types of institutions used Facebook for accessing course notes and other materials the most, public 
university students seemed to use it more than private university students. However, students at private 
universities had higher mean scores of the other three statements than those at public universities. 


Table 3. Mean and Standard Deviation of Facebook Usage for Academic Purposes 



Public University 

Private University 

Mean 

SD. 

Mean 

SD. 

Communication 





1 . Communicate with other students in my course 

4.34 

0.75 

4.26 

0.74 

2. Communicate with the teachers 

2.68 

1.00 

3.02 

1.06 

3. Communicate with university staffs 

2.38 

1.04 

2.40 

1.05 

4. Keep track of university announcements 

3.53 

1.06 

3.19 

1.10 

Sharing 





5. Exchange information with classmates 

4.11 

0.87 

4.02 

0.87 

6. Send and receive information among friends with the 
aim to complete group reports 

3.89 

0.96 

4.02 

0.84 

7. Exchange views with friends on class subject matters 

3.67 

1.00 

3.81 

0.87 

Collaboration 





8. Access course notes and other materials 

3.95 

0.93 

3.71 

0.96 

9. Participate in activities with the teachers 

2.73 

1.04 

3.00 

1.04 

10. Complete quizzes provided by the teachers 

2.83 

1.08 

3.09 

1.06 

1 1 . Participate in activities or games which are a part of the 
coursework 

2.70 

1.08 

3.05 

1.07 


From Table 4, it can be determined that Thai students used Facebook to share more than to communicate 
and collaborate. Although public university students had higher mean score of communication than private 
university students, but they had fewer mean scores of sharing and collaboration. From the statements which 
cover three aspects (communication, sharing and collaboration), the difference between public university 
students and private university students were compared using Multivariate Analysis of Variance (MANOVA) 
to the conclusion with a significance of 0.05 that the use of Facebook for academic purposes by public 
university students and private university students differed only in the area of collaboration (F = 4.255, p- 
value = 0.040) with private university students using Facebook for collaboration more than public university 
students. No significant difference was found in the use of Facebook for communication and sharing between 
the two groups (see Table 4). 
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Table 4. Comparison in the Use of Facebook for Academic Purposes between Public University Students and Private 

University Students 



Institution 

n 

mean 

SD. 

F-value 

p-value 

1 .Communication 

Public university 

219 

3.23 

0.70 

0.058 

0.811 


Private university 

241 

3.21 

0.74 



2. Sharing 

Public university 

219 

3.89 

0.81 

0.694 

0.405 


Private university 

241 

3.95 

0.69 



3. Collaboration 

Public university 

219 

3.05 

0.81 

4.255 

0.040* 


Private university 

241 

3.21 

0.84 




*p < 0.05 


3. CONCLUSION AND DISCUSSION 

Conclusion and discussion are presented as follows: 

This study revealed that Thai students used Facebook for sharing more than communication and 
collaboration. This sharing occurred among the students themselves. This is due to the fact that students feel 
more comfortable to talk with their friends than teachers, university staffs and others. They may not want the 
teachers to know much about themselves. That is why the activity they feel free to do the most in the aspect 
of collaboration on Facebook is accessing course notes and other materials as shown in Table 3. Another 
reason is probably because Facebook has not yet become popular as an instructional tool for the teachers. The 
study conducted by Moran, Seaman, and Tinti-kane (201 1) echoes that 77% of teachers use social media and 
for personal purposes. Only 60% of the teachers use social media for instructional purposes, and of this 
number, only 4% of the teachers actually use Facebook as a tool for the instruction. Teachers are not ready to 
use Facebook in their instruction because the growth in use of online social network is more evident in 
students and younger people than in elder population (Jones & Fox, 2009; Matney & Borland, 2009). In this 
case, although students would like to contact their teachers through Facebook, they cannot make it. 

When making a comparison between the two groups of institutions, the study found that public university 
students used more Facebook for communicating with other students in the course as well as keeping track of 
university announcements than private university students who used more Facebook to communicate with 
teachers and university staffs. This may be because public universities tend to have better students who are 
able to catch up with the lessons. They rarely have problems in their study. Students at private universities 
often inquire and make a request of suggestion from teachers and people concerned in case they do not 
understand the contents or information. 

However, using Facebook to access course notes is, therefore, the only one in the aspect of collaboration 
which public university students employed more than their counterparts at private universities. Since students 
at public universities are more focused on study and active, they tend to use Facebook to access course notes 
and other materials than private university students as demonstrated in Table 3. In spite of this fact, private 
university students were found to use Facebook for the purposes of collaboration more than public university 
students. Students at private universities got more involved with many collaborative activities on Facebook. 
This is possibly because of the fact that, by nature, private university students are less focused on their 
studies than public university students. Therefore, it is necessary for private university teachers to devise 
group work requiring the collaboration of students. On the other hand, the learning environment of public 
universities is more traditional with the teachers relying heavily on in-class lectures. There is no need for 
students to collaborate in their studies. 
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ABSTRACT 

The purpose of this paper is to determine whether the changes that were found to occur pre- to post intervention in 
students' cognitive structures (Mills, 2013; Knezek, Christensen, Tyler-Wood, & Periathiruvadi, 2013) continued to 
persist two years later. Major findings were: a) semantic perception of science and STEM as a career became more 
aligned with interest in being a scientist, from pretest to post test time during the treatment year and continued to be 
aligned two years later; b) semantic perception of engineering moved from alignment with science and STEM as a career 
at time 1, to alignment with semantic perception of technology and creative tendencies after the treatment year, at time 2, 
and remained aligned with technology two years later, at time 3; and c) semantic perception of mathematics was 
separated from the other constructs during the pre-post treatment year and remained largely separated two years later. 
Data mining techniques were also used to explore changes in relationships among these and other constructs over time. 


KEYWORDS 

Cognitive constructs, data mining, middle school students, STEM education 


1. INTRODUCTION 

Middle school students were the target of this hands-on energy monitoring project. The Middle Schoolers 
Out to Save the World (MSOSW) project was funded from the Innovative Technology Experiences for 
Students and Teachers (ITEST) program of the National Science Eoundation (NSE). The main goal of the 
MSOSW project has been to interest and prepare middle school students to participate in the science, 
technology, engineering and mathematics (STEM) workforce of the future. 

While there were 15 classrooms in eight schools participating in the MSOSW project, the researchers 
were able to gather longitudinal data two years after the treatment for one school (with three classrooms). 
These students were surveyed three times, once prior to the treatment (pretest) in the fall of their sixth grade 
year, once following the treatment (post test) in the spring of their sixth grade year and again two years later 
in the spring of their 8* grade year. 


2. RESEARCH STUDY 
2.1 Instrumentation 

Three different survey instruments were used to gather project data. The data were gathered online via the 
project website. The STEM Semantics Survey (Tyler-Wood, Knezek, & Christensen, 2000) was used to 
measure perceptions of science, mathematics, engineering, technology, and STEM careers, while the 
Computer Attitude Questionnaire (CAQ) (Knezek & Christensen, 1998) was used to measure self -perceived 
motivation, creative tendencies, and attitudes toward school. The Career Interest Questionnaire (CIQ) 
(Bowdich, 2009) was used to examine student attitudes toward a career in science. 

The STEM Semantic Survey is a semantic differential instrument designed to assess elementary through 
secondary students’ as well as pre-service and in-service teachers’ perceptions of STEM disciplines. 
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This instrument consists of 25 items, divided into 5 sub-scales: Science, Mathematics, Engineering and 
STEM as a Career. The items for each of the five scales are semantic adjective pairs such as 
boring/interesting or exciting/unexciting. Previous studies using this instrument (n = 174) revealed internal 
consistency reliabilities on perceptions of science, math, engineering, technology, and STEM as a career 
ranged from a = .84 to a = .93 (Tyler-Wood, Knezek, & Christensen, 2010). 

Learner disposition measurement scales from the Computer Attitude Questionnaire (Self concept. 
Creative Tendencies, and Attitudes toward School are also utilized in this study (Knezek, Christensen, 
Miyashita & Ropp, 2000). These scales are comprised of Likert-type question items with response ratings 
ranging from strongly disagree (1) to strongly agree (5). The reliabilities of these scales ranged from a = .72 
to a = .88 in a prior study (Knezek & Christensen, 2000). 

The CIQ is a Likert-type (1 = strongly disagree to 5 = strongly agree) instrument composed of 12 items in 
three sections that can each form a measurement scale. Both the 3 parts as well as the 12-item total CIQ scale 
was used in this study. The CIQ is adapted from an instrument developed a project promoting STEM interest 
(Bowdich, 2009). Reliabilities for the CIQ total scale score based on 12 items have typically fallen in the 
range of alpha = .94, while part a, b, and c reliabilities have typically ranged from .78 to .94 (Tyler-Wood, 
Knezek, & Christensen, 2010). 

2.2 Methods 


Eollow up data were collected from one set of students (n=60) two years after they participated in the 
MSOSW project activities as sixth graders. These students completed pretest surveys in fall of 2010 (time 1), 
posttest surveys in spring of 201 1 (time 2) and follow up surveys in spring 2013 (time 3). 

2.3 Findings 

2.3.1 Descriptive Findings 

Descriptive data of the significant changes in dispositions are supplied in Table 1 for all three administrations 
of the surveys. As shown in Table 1, this group of students generally followed the trend noted in previous 
studies in steady declines in learning dispositions as students progress through grade levels (and age) in 
school (Knezek & Christensen, 2000; Christensen & Knezek, 2001). One noteworthy exception is in Attitude 
Toward School that exhibited a significant increase from pretest to post test time during the MSOSW 
activities of their sixth grade year, before reverting to the general trend towards decline by the eighth grade 
year. 


Table 1. Analysis of Variance for Three Administrations of Survey Instruments 



N 

Mean 

Std. Dev. 

Sig. 

CAQ Attitude Toward 
School 

Time 1 

63 

3.06 

.64 


Time 2 

53 

3.18 

.51 


Time 3 

60 

2.90 

.60 


Total 

176 

3.04 

.60 

.039 

CAQ Self Concept 

Time 1 

63 

4.83 

1.05 


Time 2 

53 

3.96 

.63 


Time 3 

60 

3.85 

.66 


Total 

176 

4.23 

.92 

.000 

STEM Science 

Time 1 

63 

5.98 

1.11 


Time 2 

53 

4.92 

1.37 


Time 3 

59 

4.98 

1.52 


Total 

175 

5.32 

1.42 

.000 

STEM Technology 

Time 1 

63 

6.27 

1.10 


Time 2 

53 

5.89 

1.20 


Time 3 

59 

5.73 

1.41 


Total 

175 

5.97 

1.26 

.048 

CIQ Parti 

Time 1 

63 

3.21 

.79 


Time 2 

53 

2.85 

.97 


Time 3 

59 

3.12 

.94 


Total 

175 

3.07 

.90 

.087 
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2.3.2 Gender Differences 

Gender differences for the key measnres exhibiting significant (p < .05) differences at time 1 are shown in 
Table 2. As shown in Table 2, at the beginning of the sixth grade year, before taking part in MSOSW 
activities, girls reported significantly (p < .05) lower semantic perceptions of science, engineering, and 
STEM as a career than boys. Effect sizes for boys versus girls ranged from .46 for STEM as a Career to 1.06 
for perception of engineering. These lie in the range of moderate to large according to guidelines by Cohen 
(1988). They surpass the ES = .3 criteria at which the magnitude of a difference is normally considered 
educationally meaningful (Bialo & Sivin-Kachala, 1996). 

After completion of MSOSW activities, at the end of the sixth grade year in school, a very different 
picture regarding gender differences emerged. Specifically, there were no (p < .05) significant differences 
between boys and girls on any of the three measures of science, engineering, and STEM as a career. In 
addition, effect sizes had been reduced to the point at which none of the three identified measures indicated 
educationally meaningful (ES > .3) differences between the semantic perceptions of science, engineering, 
and STEM as a career for boys and girls. As shown in Eigure 1, generally the boys declined a large amount 
from pretest to posttest, while girls declined little or (in the case of engineering) became more positive in 
their perceptions from pretest time to post. This implies that the MSOSW activities had an especially positive 
impact on girls, as was reported for the overall findings across schools by Knezek, Christensen, Tyler-Wood, 
and Periathiruvadi (2013). 

At the time of the follow-up assessment two years later, at the end of the eighth grade year (time 3), girls 
had generally retained their end-of-sixth grade perceptions of science, engineering, and STEM as career 
(Eigure 1), but the boys had rebounded to the point where their dispositions toward science, technology and 
engineering were once again significantly (p < .05) more positive than girls in all three areas of semantic 
perception of science, engineering, and STEM as a career. 


Table 2. Significant Differences between Males and Females on STEM Measurement Indices 



Gender 

n 

Mean 

SD 

Sig 

ES 

STEM Science Time 1 

Male 

31 

6.30 

.95 



Eemale 

32 

5.67 

1.19 



Total 

63 

5.98 

1.11 

.024 

.59 

STEM Engineering Time 1 

Male 

31 

5.91 

1.49 



Eemale 

32 

4.30 

1.56 



Total 

63 

5.09 

1.72 

.000 

1.06 

STEM Career Time 1 

Male 

31 

5.62 

1.22 



Eemale 

32 

4.94 

1.68 



Total 

63 

5.27 

1.50 

.069 

.46 








STEM Science Time 2 

Male 

25 

5.00 

1.54 



Eemale 

28 

4.86 

1.24 



Total 

53 

4.92 

1.37 

.709 

.10 

STEM Engineering Time 2 

Male 

25 

5.18 

1.93 



Eemale 

28 

4.71 

1.37 



Total 

53 

4.93 

1.66 

.303 

.28 

STEM Career Time 2 

Male 

25 

4.64 

1.69 



Eemale 

28 

4.39 

1.42 



Total 

53 

4.51 

1.54 

.548 

.16 








STEM Science Time 3 

Male 

33 

5.32 

1.58 



Eemale 

26 

4.54 

1.33 



Total 

59 

4.98 

1.52 

.048 

.54 

STEM Engineering Time 3 

Male 

32 

5.01 

.81 



Eemale 

26 

4.10 

.97 



Total 

58 

4.60 

.99 

.000 

1.02 

STEMCareerS time 3 

Male 

33 

5.40 

1.58 



Eemale 

27 

4.55 

1.16 



Total 

60 

5.02 

1.46 

.023 

.61 
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Eigure 1. Changes Pre-Post for these Three Across Time 1, 2, 3. 

2.3.3 Correlational Relationships for Time 1, 2 and 3 

Trends in correlations that changed from time 1, to time 2 and time 3 were a primary impetus for examining 
these data from a construct-based perspective. As shown in Table 3, semantic perception of technology was 
the only STEM measure strongly associated with other gathered indicators at the time of the pretest, time 1 . 
Especially notable is the positive association between STEM technology measures and creative tendencies (r 
= .303, p < .016). By the end of the sixth grade treatment year, significant positive correlations existed 
between creative tendencies and semantic perception of engineering (r = .377, p < .005) semantic perception 
of technology (r = .275, p < .047), CIQ Part 1 (r = .275, p < .047), CIQ Part 2 (r = .327, .017), CIQ Part 3 (r 
= .285, p = .040), and the CIQ total scale score (r = .346, p < .011). Associations had developed over the 
course of the year of MSOSW activities between creative tendencies and numerous measures of interest in 
“doing science” or “being a scientist” as a career. 

Two years later, at time 3, creative tendencies had retained its significant (p < .05) associations with CIQ 
Part 1 (family and environmental influence on STEM career) (r = .263, p < .044) and on science in the form 
of semantic perception of science (r = .310, p < .017). In addition, computer enjoyment had become aligned 
with semantic perception of STEM as a career (r = .331, p < .01). These patterns indicate that the cognitive 
perceptions of the MSOSW participants underwent an extensive change from the beginning to the end of the 
MSOSW treatment activities school year. In addition, evidence shows that some of these were still retained 
two years later. 

Systematic examination of changes in these inter-correlations led to exploration of the emergence and 
retention of these constructs through higher-order factor analysis and data mining techniques. 

2.3.4 Higher Order Factor Analysis 

Higher order factor analysis is a type of exploratory factor analysis in which the variables to be factor 
analyzed are scale scores rather than individual item responses (Dunn-Rankin, Knezek, Wallace, & Zhang, 
2004). 

Each scale score typically itself represents a defined construct (such as creative tendencies or perception 
of mathematics) and therefore the rationale for the process is searching for higher-order constructs that may 
explain an individual’s attributes on several constructs related at a higher level. 

As shown in Table 3, at the time of the pretest when the students were beginning the sixth grade, the three 
parts of the CIQ were aligned in Eactor 1 together while semantic perception of engineering and semantic 
perception of STEM as a career and semantic perception of science were together in Eactor 2. Eactor 3 
contained creative tendencies and semantic perception of technology, while in Eactor 4 mathematics emerged 
as a higher order construct on its own. The four factors extracted explained 77.0% of the common variance in 
the data. Note that semantic perceptions of science and STEM as a career were not strongly aligned with the 
CIQ scales relevant to having a career as a scientist or doing science, and creative tendencies clustered with 
semantic perceptions of technology. 
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Table 3. STEM-related Constructs Existing Among 6* Grade Students as of Pretest Time of the Treatment Year 



Component | 


1 

2 

3 

4 

CIQ Part 3 

.847 



.264 

CIQ Part 1 

.816 

.411 


.135 

CIQ Part 2 

.800 

.406 


-.142 

STEM Engineering 

.306 

.781 



STEM Career 

.437 

.648 


.354 

STEM Science 

.381 

.421 

.287 

.382 

CAQ Creative Tendencies 

.115 

-.363 

.804 


STEM Technology 


.387 

.789 


STEM Mathematics 




.942 


Notes: Extraction Method: Principal Component Analysis, Rotation Method: Varimax with Kaiser Normalization, Rotation 
converged in 4 iterations. Factor loadings < . 1 suppressed 


By the end of the sixth grade year, CIQ Part 1 (family and environmental support for career as scientist) 
and Part 2 (interest in college courses in science) clustered together in Factor 1 with semantic perception of 
STEM as a career and semantic perception of science. Semantic perception of engineering joined creative 
tendencies and semantic perception of technology in Factor 2. CIQ Part 3 (making the world a better place 
through science) separated out to be on its own higher order construct, and semantic perception of 
mathematics continued as its own higher order construct, in Factors 3 and 4. The four factors extracted 
explained 75.1% of the common variance in the data. These relationships are shown in Table 4. 

Table 4. STEM-related Constructs Emerging Among 6* Graders by the end of Treatment Year (time 2) 



Component | 


1 

2 

3 

4 

CIQ Part 2 

.852 


.189 

-.138 

CIQ Part 1 

.800 

-.101 

.434 

-.143 

STEM Career 

.595 

.568 

-.182 


STEM Science 

.564 

.219 

.512 

.361 

STEM Engineering 

.227 

.774 



STEM Technology 

-.323 

.766 

.230 


CAQ Creative Tendencies 

.219 

.543 


-.537 

CIQ Part 3 

.239 

.133 

.868 

-.165 

STEM Mathematics 




.852 


Notes: Extraction Method: Principal Component Analysis, Rotation Method: Varimax with Kaiser Normalization, Rotation con^ 
in 7 iterations. Factor loadings < . 1 suppressed 


As shown in Table 6, at the time of the follow up survey two years later, CIQ parts 1 (family and 
environmental support for career as scientist) and Part 2 (interest in college courses in science) remained 
clustered together in Factor 1 with semantic perception of STEM as a career and semantic perception of 
science, while semantic perception of engineering and semantic perception of technology remained clustered 
in Factor 2. CIQ Part 3 (making the world a better place through science) also remained as its own higher 
order construct in Factor 3, while creative tendencies separated from its time 2 clustering with semantic 
perception of engineering and semantic perception of technology in Factor 2 and became the opposite pole of 
semantic perception of mathematics in Factors 4. The four factors extracted explained 78.1% of the common 
variance in the data. These relationships are shown in Table 5. 

Table 5. STEM-related Constructs Existing Among 6* Graders at the end of their 8* Grade Year 



Component | 


1 

2 

3 

4 

CIQ Part 2 

.932 




CIQ Part 1 

.870 

.117 

.301 

.103 

STEM Career 

.674 

.528 

-.169 


STEM Science 

.571 

.488 

.321 

.304 

STEM Technology 


.849 

.313 


STEM Engineering 

.366 

.756 
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CIQ Part 3 

.157 

.152 

.928 


CAQ Creative Tendencies 


.230 


.824 

STEM Mathematics 


.439 

-.145 

-.607 


Note: Extraction Method: Principal Component Analysis, Rotation Method: Varimax with Kaiser Normalization, Rotation converged 
in 7 iterations, Factor loadings < . 1 suppressed 


2.3.5 Evolution of Constructs over Time Based on Higher Order Factor Analysis 

Several changes in the conceptual frameworks of the students involved in this project can be identified as still 
persisting at the time of the two-year follow-up study. These can be identified through higher-order factor 
analysis even as the results of ANOVAs comparing increases or decreases at time 1 versus time 2 and time 3 
are inconclusive at best. The major trends can be summarized as: 

1. Desire to be a scientist (CIQ Parts 1, 2, 3) formed its own factor prior to initiation of MSOSW 
activities at the beginning of the sixth grade school year. Desire to be a scientist and the semantic perception 
of a career in STEM became attached to semantic perception of science by the end of the sixth grade 
treatment year. This alignment of desire to be a scientist with semantic perception of science continued to 
persist two years later, when the students were at the end of their eighth grade year in school. 

2. Semantic perception of science was clustered together with semantic perception of engineering (in 
Factor 2) at the beginning of the sixth grade year for these students. By the end of the sixth grade treatment 
year semantic perception of science had moved to be clustered with desire to be a scientist (CIQ Parts 1 and 
2), while semantic perception of engineering, technology, and creative tendencies clustered together to form 
Factor 3. Two years later, at the end of the eighth grade year, semantic perception of STEM as a career 
remained aligned with semantic perception of science and semantic perception of STEM career. 

3. Semantic perception of mathematics, at the time of the pretest and posttest of the sixth grade year, was 
its own Factor 4. By the end of the eighth grade year, semantic perception of mathematics was joined by 
creative tendencies but as a polar opposite meaning those who had higher perceptions of creative tendencies 
had lower semantic perceptions of mathematics. 

2.3.6 Analysis Based on Data Mining Techniques 

Using the Eureqa data mining software developed at Cornell University, three measures were compared: 
Time 1, 2 and 3 which represent a pre-assessment, a one year post and a two-year post. Searches of 1 min 30 
sec were performed on each data set, during which time approximately 2.2 * lO'^O evaluations were 
performed comparing potential symbolic expressions against a metric of minimizing the absolute error 
(Figure 2). A progress over time graph for each search represented the drop in absolute error over time and 
helped verify that a reasonable level of stability (e.g. a horizontal line) had been reached at the time of the 
conclusion of the search. 
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Figure 2. Graphic representation of a search for equations given a data set documents the time of the search, the 
performance of the computational method, and confidence metrics. 

At the conclusion of each search, a decision is made concerning which equation to select to represent the 
best trade-off of complexity versus error with an acceptable correlation coefficient. A number of comparisons 
aid in the decision, for example, the gain in correlation coefficient per unit of complexity, the change in 
number of coefficients, and the use of variables within the candidate expressions. In general, a solution is 
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selected that minimizes both error and complexity and which has an interpretable mathematical structure. 
The analysis then proceeds to find and interpret changes among the selected equations that may have 
statistical support from regression analysis, that may have external support from previous research, that may 
help further explain or validate previous results, and that may suggest additional analyses. 
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Figure 3. Equation options at the conclusion of a search showing the position (red dot in the lower right panel of the 
graphic) of the selection along the Pareto curve of complexity versus accuracy. Full solution details are reported in the 

narrative. 


Table 6. Results of Search for Best Predictors of Science Career Interest Using Eureqa Data Mining Software 


CIQTotal = f(CAQCreativeTendencies, CAQAttitudeTowardSchool, CAQSelfConcept, STEMScience, 
STEMMathematics, STEMEngineering, STEMTechnology) 


Time 1 

Time 2 

Time 3 

R'^2 Goodness of Fit 

0.52122356 

0.52367202 

0.58247764 

Correlation Coefficient 

0.72217347 

0.72538579 

0.76433976 

Complexity 

19 

21 

21 

Time 1 Solution 

CIQTotal = CAQAttitudeTowardSchool + 0.08586*CAQCreativeTendencies * 
STEMMathematics -i- 0.02497*STEMScience*STEMEngineering - 
0.1384*CAQAttitudeTowardSchool * STEMMathematics 

Time 2 Solution 

CIQTotal = 3.836*CreativeTendencies + 0.1653*CAQCreativeTendencies*STEMScience - 
4.253 - 0.3318*STEMScience - 0.5892*CAQCreativeTendencies^2 

Times 3 Solution 

CIQTotal = 1.789*CAQAttitudeTowardSchool 4- 0.3188*STEMScience -i- 
0.06987*STEMTechnology''2 - 1.364 - 
0.277*CAQAttitudeTowardSchool*STEMTechnology 


Major contributions of the Eureqa data mining analyses were: a) to reconfirm that semantic perception of 
science plays an important role in predicting reported interest in Being a Scientist, b) to reconfirm the 
important association of creative tendencies at time 1 and time 2 with interest in Being a Scientist, well as to 
highlight questions about why creative tendencies became disconnected (not a major predictor) by the end of 
the eighth grade year, at time 3; and c) to note that attitude toward school was a major predictor of interest in 
Being a Scientist at time 1, was not a major predictor at time 2, and then returned to be a major predictor at 
time 3. Eureqa findings collectively imply that the hands science activities introduced during the sixth grade 
may have ameliorated the well-known trends toward decline in attitudes toward school as students advance 
through higher grade levels. Eurther research is needed in this area. 


3. CONCLUSION 

Semantic perception of Science and of STEM as a Career became more closely aligned with Doing Science 
and Being a Scientist over the course of the MSOSW grant activity year. This alignment remained in place 
two years later. Two major outcomes of the MSOSW project were related to the current study. 
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First, students emerge from project activities with increased aspirations for STEM-related careers with 
gains in students’ understanding of what scientists do, and with gains in belief that scientists can work on 
things that help the world. Second, female students gained more than males in their perceptions of science, 
engineering, mathematics and technology (Knezek, Christensen, Tyler-Wood & Periathiruvadi, 2013). The 
current findings provide further evidence that the positive impact of MSOSW persists in student participants 
for several years beyond the time frame of the project. This finding is consistent with a nine-year follow-up 
study of BUGS girls (Tyler-Wood et al, 2011) that used the same instruments employed in the current study 
to demonstrate that participants’ higher STEM dispositions (vs. childhood comparison group) persisted from 
elementary school into university studies. 
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ABSTRACT 

E-learning content development can be regarded as an example of knowledge delivery process because the developers 
have to receive knowledge transferred from subject-matter experts (SMEs), to translate the received knowledge with 
appropriate instructional design, and to transform the project outcomes to fulfill learner’s learning needs. As developers 
and SMEs may come from different disciplines, using distinct terminologies, having inconsistent interpretations, and 
holding different values, however, problems of knowledge boundaries may arise and prohibit a successful knowledge 
delivery process. This study conducts a case study to explore the problems caused by knowledge boundaries between 
developers and SME in e-leaming content development. Our findings illustrated how the syntactic, semantic, and 
pragmatic knowledge boundaries caused the misunderstanding problems which inhibited the effective knowledge 
delivery. Our findings also concluded the activities and boundary objects that the developers used to bridge the 
knowledge boundaries. 


KEYWORDS 

E-learning content development, knowledge boundary, knowledge boundary spanning, boundary objects, case study. 


1. INTRODUCTION 

More and more organizations emphasize to leverage knowledge for increasing profits (Brown & Duguid, 
2001; Hooff & Bidder, 2004). Therefore E-learning has become one of the mostly used instruments for 
knowledge deliveries and employee training (Rosenberg, 2006). To develop better e-learning contents, 
developers have to understand the domain knowledge with the aid of subject-matter experts (SMEs) so that 
they are able to select appropriate instructional methods and media presentations for developing e-learning 
contents. Therefore, e-learning content developments can be regarded as the knowledge delivery process 
between SMEs and developers. 

However, knowledge deliveries between SMEs and developers can be ineffective because of knowledge 
boundaries among professional domains. Since the SMEs and developers come from different professional 
domains which share inconsistent interpretations and hold different values, their communications may not be 
on the same page, and misunderstandings may be arisen when they collaborate (Boland & Tenkasi, 1995; 
Carlile, 2002; Dougherty, 1992). Eor exploring the knowledge boundary problems, this study adopts a case 
study in e-learning content development context for seeking answers for two research questions: (1) What are 
the knowledge boundary problems encountered in an e-learning content development project? (2) How can 
these knowledge boundaries be spanned and solved? 
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2. LITERATURE REVIEW 

2.1 Knowledge Boundary and Boundary Spanning 

Knowledge boundaries challenge effective knowledge deliveries across professional boundaries (Brown & 
Duguid, 2001; Carlile, 2002, 2004). When Knowledge is localized and embedded in practice (Bourdieu, 
1980; Lave, 1988), knowledge of certain functions is not to fit into the “living world” of another (Yanow, 
2004). This specialization of knowledge in practice makes it difficult to collaborate across professional 
boundaries, as well as to accommodate the knowledge developed in other practices (Carlile, 2002). 

Carlile (2004) proposed an integrated framework to manage knowledge boundaries (Figure 1). In the 
framework, knowledge boundaries differ from each other in terms of the degree of difference, dependence, 
and novelty (Carlile 2004). When the degree of novelty, dependence and specialization are low, the difficulty 
in knowledge delivery comes from incompatible terminologies which form syntactic boundary (Carlile, 
2004). It is important to establish a common lexicon for actors to transfer accurate communications and solve 
information processing problems (Davenport and Prusak, 1998; Carlile, 2002). When the degree of novelty, 
dependence and specialization increase, members from different disciplines may have different 
interpretations to the common terminologies, thus it results in semantic boundary (Carlile, 2002). It is 
because individuals use a word in different meanings in functional settings (Dougherty, 1992). To establish 
shared meanings for ensuring accurate interpretations of the knowledge across boundaries can work through 
these semantic differences (Carlile, 2002, 2004; Dougherty, 1992; Nonaka, 1994). Moreover, when the 
degree of difference, dependence and novelty are high, conflicts among the actors will show up when the 
goal of knowledge delivery contradicts with each other and then result in the pragmatic boundary (Cable, 
2002; Feng et al, 2010). To resolve the possible negative consequences of goal conflict, individuals have to 
alter their own knowledge and become capable of transforming the knowledge used by other functions for 
establishing a common interest of the project (Cable, 2002). 

Types of boundaries 



/ 

Increasing 

novelty 


Figure 1. Framework of Knowledge Boundary (Adapted from Cable (2004)) 


2.2 Boundary Objects 

Boundary objects are artifacts that provide communicative aids for individuals to exchange information with 
their counterparts (Bechky, 2003; Carlile, 2002; Star & Griesemer, 1989). Examples of the boundary objects 
can be medical records, product prototypes, engineering drawings, and standard operating procedures 
(Bechky, 2003; Carlile, 2002; Star & Griesemer, 1989). 
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Boundary objects are important for spanning the three knowledge boundaries (Carlile, 2004). As for 
syntactic boundary, a common lexicon which sufficiently specifies the differences at the boundary can 
function as a boundary object to transfer knowledge between senders and receivers (Carlile, 2004). As for 
semantic boundary, translating knowledge is a must for actors to grasp and convey the actual meaning of 
knowledge delivered from counterparts. Cross-functional team (Ancona and Caldwell, 1992), shared tools or 
methodologies, boundary spanners (Hargadon and Sutton 1997), and community of practices (Lave and 
Wenger 1991; Brown and Duguid, 1991) can be ways of developing shared meanings (Carlile, 2004). For 
spanning the pragmatic boundary, common interests have to be developed (Carlile, 2004). Shared artifacts, 
that act as boundary objects have been proved effective in providing concrete means of representing different 
interests and facilitating negotiations and transformations (Cable, 2004). 


3. RESEARCH METHODS 

3.1 Case Selection 

This research adopted a case study to explore knowledge boundary problems and boundary spanning in an 
e-learning content development context. The research was anchored in a project that developed e-learning 
materials for Chinese Puppetry. Based on theoretical sampling, the case was selected because of three 
reasons: Firstly, this case has interdisciplinary collaborations involved where the developers and the SMEs of 
Chinese Puppetry had to work together for developing the e-learning project. Secondly, the developers’ 
knowledge was quite different from that of the SMEs. Thirdly, the observed phenomenon of this case aligned 
with our research issues. 

3.2 Data Collection and Analysis 

The data collection in this study was through both archives and interviews. We collected project documents 
such as e-mails and design documents in order to know the project development details and results. In 
addition, we interviewed the developers and SMEs for understanding their experience and thoughts on 
knowledge boundary. There are seven interviews, each of which listed about 45 minutes to 2 hours of length, 
and each of them was recorded and transcribed. As for data analysis, we read the transcripts sentence by 
sentence, and divided them into meaningful phases, and then gave each phase an appropriate label to 
represent the concepts. Next, the labeled concepts were included into the schema collected from previous 
literatures and formed the theme of the interpretation. Einally, it came out of three scenarios to explain the 
research findings. 


4. RESEARCH EINDINGS 

4.1 Scenario 1: Bridge the Gap of Knowledge Terminology 

The knowledge delivery problems between the developers and SMEs were found in the beginning of the 
project. In order to design a learning content, developers needed to know what Chinese Puppetry was. They 
had to acquire the knowledge before they were able to design the contents and instructions that might be 
needed. When they did research on puppetry knowledge, however, they found that puppetry was not as 
simple as they originally thought. A developer described her surprise: 

“We were surprised to find out that roles have five criterion in Chinese Puppetry: male leads ‘Shen’, 
female leads ‘Dan’, supporting male leads ‘Jing’, an minor supporting roles ‘Mo’ and jesters ‘Chou’. Each 
role is distinguished with its own posture and paces. ... Eor example, a scholar walks slower and his 
movement is relatively small, and a warrior swaggers. These five main characteristics complete the role-plays 
in Chinese Puppetry.” 
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What’s more complicated was that there had been many factions of Chinese Puppetry, each of them had 
its unique ways of performing. Thus, the complexity of Chinese Puppetry knowledge was increased. A 
developer explained that: 

“Now, each faction has its own unique performance style, presenting by different accents in aside, as well 
as story plots.” 

The developers and SMEs hardly had experiences in each other’s expertise fields. The lack of common 
terms in the beginning of collaboration makes the developers and SMEs spend a lot of time explaining 
terminologies. Eor this, SMEs offered books that systematically explained the differences of different puppet 
criterions in different factions to make better understanding of Chinese Puppetry’s history. A developer 
mentioned that: 

“...The books had been organized by Chinese Puppetry experts and we are provided with a clear 
framework of Chinese Puppetry. There exists Chinese Puppetry terminologies with detailed annotations. 

Eor better team works, SME offered books and hand drawings to help the team acquiring basic 
terminologies of Chinese Puppetry. This makes a more complete and more efficient way of transferring 
Chinese Puppetry knowledge, thus a stable and shared lexicon for ensuring the accuracy of their 
communications was established. 

4.2 Scenario 2: Eliminate the Differences of Interpretation 

When the developers began to understand more about the terms in puppetry, they found that their 
“interpretations” of these terms were often incorrect. Eor example, the developers got to know how Chinese 
puppet roles were represented by interpreting different gestures and tones in a puppetry show. A developer 
described her reflection: 

“In a show, a role is distinguished through its voices and tones. Eor example, gentle voices are using to 
present scholars while deep and suppressed voices are for warriors.” 

Additionally, since the stage had limited space, a show was usually performed by one or two puppet- 
masters. That was to say, one single puppet-master might had to play several roles at the same time. And 
puppet shows would become more dynamic and complicated. A SME explained: 

“In puppet performances, puppet-masters require to distinguish the different roles at the same time by 
making the best uses of his voices. ... As a result, typical casts of Chinese puppet shows are composited by 
few characters with significant differences in performing styles.” 

And, to literally express such a tacit knowledge was difficult. A developer mentioned that it was hard to 
understand what SME wanted to say, she said: 

“To understand the various puppetry characters with the change of tones and was very difficult. SME 
tried hard to explain what he meant through colloquial description to let us grasp the meaning more clearly.” 

In order to better understand the meanings of terms, the developers began to attend puppet shows. A 
developer explained: 

“In our free time, we watched puppet shows for better interpreting the context. ...The puppet-masters 
instructed us various ways of puppet manipulations. Eor example, ‘Shen’ walks with more spirit, while ‘Dan’ 
walks tenderly. We also visited a SME’s studio to better observe the making of Chinese puppets. The SME 
explained every detail for us.... Eor example, different characters would have distinctive facial features with 
different performing styles required” 

With the effort spent as mentioned above, the team tried the best to understand Chinese puppetry and to 
ensure correct knowledge deliveries. These processes translated tacit knowledge and established a stable 
understanding of Chinese puppetry to ensure that there was no misunderstanding in knowledge delivery. 

4.3 Scenario 3: Negotiate Different Expectations of Project Results 

Expectations difference was found when the developers demonstrated a first prototype to SME. They found 
that the prototype was quite different from what the SME had in his mind. And, the scope of the project was 
adjusted along with the interactions between the developers and SMEs. In the beginning of the project, the 
developers attempted to present a general introduction to puppetry, but they found that it would include a 
large amount of disorganized information after they consulted the SMEs. When they interacted with SMEs, 
the developers found that SMEs were worried about the absence of young audiences. This caution thus 
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shifted the developers’ focus from developing the learning content for general public to doing that for 
attracting the youth. A developer mentioned that: 

“We were told by SMEs about less young people are attracted by puppet show. . . . We also found that 
young people would not be interested in the existing puppetry websites which provided amount of 
information with texts and pictures. ... Finally, we decided to decrease the project scope and to present the 
digitalized contents in an interactive way in order to make our e-learning content more interesting and 
attractive to the youth.” 

On the other hand, SMEs had experienced adjustments of their own expectations, too. The SMEs, experts 
of puppetry used to understand Chinese puppetry by characters. In their minds, the project should represent 
Chinese Puppetry by five typical criterion: ‘Shen’, ‘Dan’, ‘ling’, ‘Mo’ and ‘Chou’ to make audiences easily 
recognize the core of a puppet show. For this, a SME explained: 

“...we expect to represent the Chinese Puppetry according to characters. When we watch a puppet show, 
we are interested in the five roles that a Chinese puppet represents. We can recognize the roles by the 
particular appearances, clothes, and tones performed by puppets. 

When they worked with the developers, SMEs altered their expectation of the project from “being 
professional” to “being interesting.” They agreed with the developers to simply the project to alter the 
“abstruse” stereotype held by the youth to Chinese puppetry, therefore the project could be interesting to 
arouse the youth’s interest. A SME explained that: 

“Knowledge presented in a professional way is much like a textbook written in a foreign language, which 
will not appeal to the public. It is too hard for them to understand the contents. It therefore keeps distance to 
the general public. Instead, introducing puppetry in the way as the team proposed may be better. The 
knowledge becomes friendly and more attractive, which might arouse people’s interest.” 

The pragmatic knowledge boundary rooted in the fact of incongruent expectations between developers 
and SMEs. While SMEs considered the project to be comprehensive and professional, the developers wanted 
the audience to become familiar with the knowledge. The team would not be aware of these inconsistent 
expectations until the developers present a prototype to SMEs. The developers and SMEs modified their 
expectations along with rounds of revised prototypes, scripts and sketch that worked as boundary objects for 
consolidate the interests of the two party. 


5. DISCUSSION 

This study explores the knowledge boundary problems and boundary spanning in e-leaming content 
developments. Table 1 presents the research findings. 


Table 1. Knowledge Problems, Boundary Spanning Activities and Boundary Objects 



Knowledge problems 

Boundary spanning activities 

Boundary objects 

Scenario 1: 

• Overload with professional 

• Consulting SME and taking notes 

• Books 

syntactic 

terms 

• Reading books recommended by SME 

• SME's hand drawing 

boundary 

• Mix up factions of puppetry 

• Surfing the Internet 


Scenario 2: 

• Having incorrect 

Translate tacit knowledge by 

• Puppets shows 

semantic 

interpretations of terms 


• Puppets 

boundary 

• Hard to exchange the tacit 
knowledge 

• Engaging in puppet shows 

• Talking to puppetry artists 

• Watching experts' demonstration 

• Stage and properties 

Scenario 3: 

• Having different expectations 

• Negotiating the purpose of the 

• Prototype 

pragmatics 

to the project scope, content. 

project 

• Script Character' sketch 

boundary 

and presentation 

• Adjusting the content to be included 

• Using prototypes, scripts and sketch 
for making consensus 
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First, the syntactic boundaries made it difficult for SMEs to describe Chinese puppetry to the developers, 
and the developers failed to understand what SMEs said. To span this boundary, the developers and SMEs 
exchanged a number of professional terms in order to transfer knowledge. By the aids of books and hand 
drawing illustrations, developers gradually established common lexicons with SMEs for receiving Chinese 
puppetry knowledge. Therefore, books and hand drawing illustrations acted as boundary objects for 
successful knowledge transfers. In this light, these boundary objects collectively played the role as a “map”, 
which led the developers on the journey of doing the project concerning Chinese puppetry. Second, despite 
having common lexicons, the inconsistent interpretations prohibited a correct comprehension of Chinese 
puppetry. The developers spanned the semantic boundary by engaging themselves in Chinese puppet shows. 
Through observing what a Chinese puppet show was made and how puppet-masters presented the role of 
puppets, the developers started to get a better understanding of Chinese puppetry terms within the practical 
context. In this period, the boundary objects were puppet shows, puppets, stage and backdrops. These objects 
played the role as a “showcase,” which helped the developers to master the Chinese puppetry in practice, 
therefore enabling the knowledge translation. Third, the different knowledge backgrounds between SMEs 
and the developers had caused expectations toward the project differ. The purpose and contents of the project 
were adjusted by the negotiations between the two parties. In this period, the prototype, scripts and sketches 
played an important role as a “catalogue,” that transformed the expectations of both parties to reach the 
consensus for spanning the pragmatics boundary. 


6. CONCLUSIONS 

This research explores the boundaries made by different knowledge backgrounds in interdisciplinary 
collaborations and tries to illustrate solutions for spanning knowledge boundaries in e-leaming content 
development context. Our findings firstly describe three scenarios of knowledge problems in syntactic, 
semantic and pragmatics boundaries, followed by demonstrating boundary spanning activities and boundary 
objects for solving the boundary problems. This study contributes both academically and practically by 
bringing attention to the problems caused by knowledge itself in the process of e-learning content 
development. This study also illustrates the problems caused by knowledge boundaries in e-learning content 
development, and provides possible solutions to overcome these knowledge boundary problems. 
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ABSTRACT 

During the last decade. Open Educational Resources (OERs) have gained increased attention for their potential to support 
open access, sharing and reuse of digital educational resources. Therefore, a large amount of digital educational resources 
have become available worldwide through web-based open access repositories which are referred to as Learning Object 
Repositories (LORs). These resources have the potential to facilitate teachers to improve and enhance their day-to-day 
teaching activities. On the other hand, it has been identified that teachers could benefit from their participation in 
communities of best teaching practices by sharing, not only educational resources, but also learning designs that represent 
their pedagogical approach. As a result, there is an increased interest for the development of web-based repositories that 
facilitate open access to both educational resources and learning designs. However, the process of developing and sharing 
learning designs through web-based repositories requires authoring tools that can represent learning designs in a 
machine-readable way. A commonly accepted way that provides a standard notation language for the description of 
learning designs is the IMS Learning Design (LD) Specification. Consequently, a number of IMS LD compatible 
learning design authoring tools has already been developed. Nevertheless, most of these tools (a) are not supporting the 
process of importing and editing learning designs and (b) they are stand-alone tools restricting their widespread use. To 
this end, in this paper we present a web-based learning design authoring tool that aims to overcome the previous 
identified problems. 


KEYWORDS 

Learning design repositories, learning design tools, IMS learning design, graphical tool 


1. INTRODUCTION 

Over the past years, the term Open Educational Resources (OERs) has been emerged, aiming to promote 
open access to digital educational resources that are available online for everyone at a global level (Caswell, 
Henson, Jensen and Wiley, 2008). In response to this emerging trend, a large amount of digital educational 
resources have become available worldwide through web-based open access repositories which are referred 
to as Learning Object Repositories (LORs) (McGreal, 2008). These resources have the potential to facilitate 
teachers to enhance and improve their day-to-day teaching activities. 

On the other hand, it has been recognized that teachers could improve the quality of their teaching and 
support their motivation for enriching their teaching practices through their participation in communities of 
best teaching practices, which facilitates them to share, not only digital educational resources, but also 
learning designs (LDs) that represent their pedagogical approach (Conole, 2008). More specifically, teachers 
are able through their participation to communities of teaching practices to: (a) search and download best 
teaching practices for share and re-use, (b) discuss and collaborate about best teaching practices and (c) 
provide their feedback about the actual use of a teaching practice via ratings and/or comments (Galley, 
Conole, Dalziel and Ghiglione, 2010). As a result, there is an increased interest for the development of web- 
based repositories that facilitate open access to both educational resources and learning designs (Sampson 
Zervas and Sotiriou, 2011; Paquette, Marino, Lundgren-Cayrol and Leonard, 2008). 

Nevertheless, the process of developing and sharing LDs through web-based repositories requires 
authoring tools that can represent the pedagogical design that is followed in a typical classroom, that is, the 
structured flow of learning activities populated with resources and facilitated by certain tools and devices, 
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where teachers and students participate assuming certain roles (Griffiths and Blat, 2005; Wilson, 2005). A 
key specification that provides a standard notation language for the description of learning designs is the IMS 
Learning Design (LD) Specification (IMS Global Learning Consortium, 2003). Therefore, during the last 
years a number of IMS LD compatible learning design authoring tools have been developed (Griffiths and 
Liber, 2008. However, most of these tools: (a) are not supporting the process of importing and editing LDs, 
and (b) are stand-alone tools restricting their widespread use. To this end, in this paper we present a new 
learning design authoring tool, namely the ASK Learning Designer Toolkit 2.0 (ASK-LDT 2.0) that aims to 
overcome the previous identified problems. 

The rest of the paper is structured as follows. In section 2, we introduce the IMS LD specification and we 
present its conceptual structure. Moreover, we present and compare existing LD authoring tools, so as to 
identify their weaknesses. In Section 3, we present the architectural components and the fundamental 
functionalities of the ASK-LDT 2.0. Section 4 presents users’ satisfaction results from the use of ASK-LDT 
2.0. Finally, we discuss our findings and we present our conclusions and suggestions for future work. 


2. BACKGROUND 

2.1 IMS Learning Design Specification 

IMS LD Specification was developed by IMS Global Learning Consortium (IMS GLC) in 2003, with the aim 
of providing a standard notation language for describing LDs (IMS Global Learning Consortium, 2003). A 
learning design (LD) is defined as: “the description of the teaching-learning process, which follows a specific 
pedagogical strategy or practice that takes place in a unit of learning (e.g., an educational course, a learning 
activity or any other designed learning event) towards addressing specific learning objectives, for a specific 
target group in a specific context or subject domain” (Koper and Olivier, 2004, p. 98). IMS LD specification 
consists of the following core elements (Koper and Olivier, 2004): 

■ Activity: Activities are one of the core structural elements of the LD and they are used to express 
actions that learners or teachers perform during learning and teaching. They also specify their 
termination conditions and the actions to be taken upon termination. There are two basic types of 
activities: Learning Activities and Support Activities. A Learning Activity is directed at attaining a 
learning objective per individual actor. A support activity is meant to facilitate a role performing one 
or more learning activities. 

■ Role: specifies the participating roles in a learning/support activity. There are two basic Role types: 
the Learner Role and the Support Role. These roles can be sub-typed to allow learners to play 
different roles in certain types of learning/support activities. Similarly, support role can be sub -typed 
and given more specialized roles, such as tutor, teaching Assistant, mentor, etc. Thus, roles set the 
basis for multi-user models of learning and teaching. The name that a certain role is given depends 
on the underlying teaching approach and the setting in use. 

■ Environment: Environments are elements which hold educational resources and/or tools/services 
(such as a chat, a forum, etc.) 

■ Property: Properties are elements that store different kinds of data, which can be displayed and 
updated/changed during the teaching/learning process. Properties can be used for building 
conditions of IF-THEN-ELSE statements that control the visibility of elements such as activities and 
environments, as well as for updating of existing properties. 

The IMS LD specification follows the metaphor of a theatrical play. This means that the learning process 
is represented as a play including a sequence of acts, with each act containing a number of role parts that 
connect the roles to the learning activities they perform and to the educational resources they use (Koper and 
Olivier, 2004). In IMS LD, a learning design can be built at three (3) different levels, as follows (Koper and 
Olivier, 2004): 

■ Level A: contains a series of learning activities, performed by one or more actors/roles, in an 
environment consisting of educational resources and/or tools/services. 

■ Level B: adds properties (storing information about a person or a group), and conditions (placing 
constraints with rules upon learning flow). 

■ Level C: adds notifications that can facilitate reconfiguring design based on run-time events. 
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2.2 Learning Design Authoring Tools 

During the last years, a number of IMS LD authoring tools have been developed, which could be summarized 
below: 

■ LAMS (Learning Activity Management System) (Dalziel, 2003) is an open-source web-based 
graphical authoring tool, which enables its users to graphically design LDs of pre-defmed 
learning/support activities (users can not define new types of learning/support activities). LAMS 
supports (a) pre-defined role types and (b) pre-defmed environments integrated with specific 
specific tools/services, whereas it can not support definition of properties and conditions. 
Finally, LAMS enables export of LDs in IMS LD level A compatible format. 

■ ASK Learning Designer Toolkit (ASK-LDT) (Sampson, Karampiperis and Zervas, 2005) is a 
stand-alone graphical authoring tool, which enables its users to graphically design LDs based on 
the interconnection of user defined learning/support activities. ASK-LDT supports (a) user- 
defined role types, (b) user-defined environments and (c) definition of properties and conditions. 
Finally, ASK-LDT enables export of LDs in IMS LD level A, B compatible format 

■ Recourse (Griffiths, Beauvoir, Liber and Barrett-Baxendale, 2009) is an open source stand- 
alone authoring tool, which combines form-based and graphical-based authoring of LDs based 
on the interconnection of user defined learning/support activities. However, the dominant 
approach for authoring LDs within the tool is based on filling forms. ReCourse supports (a) user- 
defined role types, (b) user-defined environments and (c) definition of properties and conditions. 
Finally, ReCourse supports import and export of LDs in IMS LD level A, B and C compatible 
format. 

■ OpenGLM (Neumann and Oberhuemer, 2008) is an open source stand-alone graphical 
authoring tool, which enables its users to graphically design LDs based on the interconnection of 
user defined learning/support activities. OpenGLM supports (a) user-defined role types, (b) user- 
defined environments and (c) definition of properties and conditions. Finally, OpenGLM enables 
export of LDs in IMS LD level A and B compatible format. 

■ CADMOS (Katsamani, Retails and Boloudakis, 2012) is a stand-alone graphical authoring tool, 
which enables its users to graphically design LDs based on the interconnection of user defined 
learning/support activities. CADMOS supports (a) user-defined role types, (b) user-defined 
environments and (c) definition of properties and conditions. Finally, CADMOS enables export 
of LDs in IMS LD level A, B compatible format. 

Table 1 compares existing LD authoring tools according to their functionalities. As we can notice from 
Table 1, most of the existing LD tools are stand alone and they are not accessible via a web-browser. On the 
other hand, most of them can support the definition of the main IMS LD elements as identified in section 2. 1 
and only LAMS come short into supporting all main IMS LD elements. Nevertheless, it seems that only 
ReCourse can support the process of importing LDs in IMS LD compatible format. This creates an extra 
barrier to the interoperability between different existing LD authoring tools. Therefore, in this paper we 
propose a new tool namely, ASK-LDT 2.0, which is a web-based graphical authoring tool that fully supports 
the (a) process of authoring LDs by utilizing the main IMS LD elements, as well as (b) the process of 
importing and exporting LDs in IMS LD compatible format. 


Table 1. Comparison of Existing LD Authoring Tools 


Functionalities 

LAMS 

ASK-LDT 

Recourse 

OpenGLM 

CADMOS 

Stand-alone/Web Based 

Web-based 

Stand-alone 

Stand-alone 

Stand-alone 

Stand-alone 

F orm-based/Graphical- 
based 

Graphical- 

based 

Graphical- 

based 

Form-based 

Graphical- 

based 

Graphical-based 

User-defined Roles 

Pre-defined 


Z 

Z 

Z 

User-defined 
Lear ning/S uppor t 

Pre-defined 


Z 

Z 

Z 

Activities 

User-defined 

Environments 

Pre-defined 

Z 

Z 

Z 

Z 

Create/Edit 






Properties/Conditions 

- 

Z 

Z 

Z 

Z 
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IMS LD Level 
Compatibility 
Import IMS LD Package 
Export IMS LD Package 


A 

A, B 

A, B, C 

A, B 

A, B 

_ 

_ 

✓ 

_ 

_ 







3. THE ASK LEARNING DESIGNER TOOLKIT 2.0 (ASK-LDT 2.0) 

ASK-LDT 2.0 is an open source web-based graphical tool fully supporting the process of authoring learning 
designs that are compatible with IMS LD. ASK-LDT 2.0 can be used by teachers and/or educational 
practitioners, so as to graphically design LDs based on the interconnection of user-defined learning/support 
activities. Moreover, ASK-LDT 2.0 supports (a) user-defined role types, (b) user -defined environments and 
(c) definition of properties and conditions. Finally, ASK LDT 2.0 enables import and export of LDs in IMS 
LD level A, B compatible format. Figure 1 presents an overview of ASK-LDT 2.0 architecture. 



Figure 1 . ASK-LDT 2.0 Architecture 


As shown in Figure 1, the lowest level includes a repository where the LDs that are created or imported to 
the tool by its users are stored. This level includes also a relational database where information about the 
users of the tool, as well as information about the graphical representation of the LDs is stored. The lower 
level module (namely, the Data Access Object) provides access to the LDs repository, as well as to the 
relational database and includes all the necessary procedures for reading and writing to them. The next level 
includes the following modules: 

■ The Learning Design (LD) Validation Engine, which includes all functionalities relating to (a) 
validating and importing LDs to the tool compatible with IMS LD and (b) validating and exporting 
the LDs that are developed by the users of the tool based on the IMS LD specification. 

■ The Learning Design Authoring Engine, which includes all functionalities for authoring LDs based 
on the main IMS LD elements as described in section 2. 1 

Finally, the top level of the tool is the Graphical User Interface Module, which is responsible for the 
graphical representation and visualization of the LDs and the interface with the users. 

The main functionalities of ASK-LDT 2.0 can be summarized as follows: 

■ Create/Edit/Import a LD: The user has the capability to create a new LD from scratch or open and 
edit an existing LD (Figure 2). Another option for the user is to import to the tool an existing LD 
package, which is conformant with IMS LD specification (Figure 3). This import functionality is a 
unique feature not supported by other similar tools and it can highly facilitate the interoperability 
with other similar tools. 
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■ Define Roles/Environments: The user has the capability to create/edit different roles for a LD 
(learner and/or support roles as defined in IMS LD specification) (Figure 4). Moreover, the user has 
the capability to create/edit environments (as well as tools and educational resources that support 
this environment), where learning/support activities of a LD can take place. It should be noted that 
the tool supports the creation of environments that are supported by educational resources and 
pre-defined tools/services as defined in IMS LD specification, namely a forum, an announcement 
service, a chat and a send mail service (Figure 5). 
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Figure 4. Create Learner and/or Support Roles Figure 5. Define an Environment, as well as 

Tools and Educational Resources that 
support this Environment 


■ Graphically Design Activities: The user has the capability to graphically design by using the drag 
and drop functionality the flow of the learning/support activities of a LD (Figure 6). Moreover, for 
each learning/support activity the user can add description and educational resources, assign roles 
and assign environments where the learning/support activity will take place. Finally, for each 
activity the user can define properties and conditions for the completion of a learning/support 
activity or for skipping/showing the next learning/support activity of a LD. 



Figure 6. Designing the Learning Flow o a Learning 
Design 
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■ Validate/Save/Export a LD: The user has the capability to validate a LD before saving it and 
exporting it as IMS LD file package (Figure 8). Finally, the LDs are stored to a local repository and 
they can be retrieved by using an auto complete paginated search mechanism (Figure 9). 
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Figure 8. Validate and Export a LD as an Figure 9. Search LDs from ASK-LDT 2.0 
IMS LD Package Local Repository 


4. PRELIMINARY EVALUATION 

In this section, we present an experiment of using ASK-LDT 2.0 for the development of a LD. The main 
objective that we aim to address through this experiment is to measure users’ satisfaction for authoring LDs 
by using the functionalities that were identified in Section 2.2. The evaluation method we have used is the 
method of survey (Cohen, Manion and Morrison, 2007). Surveys are appropriate evaluation methods for 
measuring the attitudes and the opinions of users (Rosier, 2002). Thus, we consider this evaluation method 
suitable for our evaluation objective. 

4.1 Participants and Workshop Setup 

The study was conducted with MSc students (N=35) during their third semester of studies as a workshop, 
which was presented in the MSc Course on e-Learning at the Department of Digital Systems of the 
University of Piraeus, Greece. The workshop lasted three (3) hours and participants were asked to develop a 
LD by using the ASK-LDT 2.0. The LD was the same for all participants and it was provided to the 
participants as a document describing in details: (a) the learning flow of the learning/support activities of the 
LD and their interconnection, (b) the participating roles to each learning/support activity, (c) the 
environments where each activity should take place and (d) the properties and the conditions for completing 
each activity of the LD. 

The procedure that was followed for the evaluation workshop was the following: 

■ ASK-LDT 2.0 Introduction: the workshop started with a demonstration about ASK-LDT 2.0 to be 
used by the participants. The main objective of the demonstration was to familiarize participants 
with the functionalities of the tool, and to provide guidance for the subsequent hands-on LD task. 

The demonstration took the form of a one (1) hour presentation supported with slides including (a) a 
brief introduction of the ASK-LDT 2.0 functionalities and guidelines for using them during the LD task 
and (b) an example of developing a pre-defmed LD. 

■ LD Authoring Task: each participant was assigned the task of developing the pre-defmed LD using 
the ASK-LDT 2.0 and without any further assistance. Our intention was to ask the participants to 
develop a rather complex LD, so as to ensure that the participants had to perform more than a simple 
transfer of identical actions from the previously presented example to the task. 

■ Post task questionnaire: after the LD development task was completed, participants were asked to 
fill out a post-task questionnaire, which aimed to collect information about participants’ satisfaction 
concerning the use of ASK-LDT 2.0 during the LD authoring process. For each one of the questions 
presented in the questionnaire, a five-point likert scale was used where 5 denotes “very satisfied” 
and 1 denotes “not at all satisfied”. 
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4.2 Results 

This section presents quantitative data analysis results for participants’ satisfaction related with the use of 
ASK-LDT 2.0 functionalities for authoring LDs based on their responses to the post-task questionnaire. 
Table 2 presents the mean and the standard deviation for each question of the questionnaire that was filled by 
the participants of our study (N=35). 


Table 2. Users’ Satisfaction per Functionality 




Mean 

(N=35) 

Standard 

Nr 

Questions 

Deviation 

(N=35) 

1 

Was it easy to create learning and/or support 
activities? 

4.60 

0.28 

2 

Was it easy to graphically design the flow of the 
leaming/support activities of the pre-defined LD? 

4.48 

0.29 

3 

Was it easy to define roles and assign them to 

4.61 

0.27 


learning and/or support activities? 

Was it easy to define environments and assign 



4 

which learning and/or support activities will take 
place in these environments? 

Was it easy to define properties and conditions of 

4.53 

0.30 

5 

for showing/hiding learning and/or support 
activities of the pre-defined LD? 

4. 43 

0.30 


As we can notice from Table 2, participants’ satisfaction has achieved very high mean score for the 
examined functionalities. Additionally, the standard deviation indicated that individual participants’ 
responses are close to the participants’ satisfaction mean proving their validity for the functionalities that 
were examined via the post-task questionnaire. These results provided us with indications that ASK-LDT 2.0 
can strongly support the process of authoring learning designs and it can highly satisfy its users during this 
process. 


5. CONCLUSIONS AND FUTURE WORK 

In this paper, it was argued that there is a growing trend for the development of web -based repositories that 
facilitate open access not only to educational resources but also to LDs. It was also identified that existing 
LD authoring tools that could be used for developing and sharing LDs through web-based repositories (a) are 
not supporting the process of importing and editing LDs and (b) they are stand-alone tools restricting their 
widespread use. Therefore, we presented ASK-LDT 2.0 a web based LD authoring tool, which is based on 
IMS LD specification and it aims to overcome the limitations of existing LD authoring tools. Preliminary 
evaluation results from the use of ASK-LDT 2.0 provided us with indications that ASK-LDT 2.0 can strongly 
support the process of authoring LD and it can highly satisfy its users during this process. 

Future work includes further development of the ASK-LDT 2.0 to support: (a) Level C of the IMS LD 
specification, (b) integration for searching/sharing the developed LDs from/to existing web-based 
repositories of LDs, (c) integration for searching/retrieving educational resources from existing LORs, so as 
to be used to LDs developed by the tool and (d) incorporation of new tools (in the form of widgets) to the 
environments that are defined for supporting the learning/support activities of a LD. 
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ABSTRACT 

This paper presents a new approach to automated identification of human emotions based on analysis of body 
movements, a recognition of gestures and poses. Methodology, models and automated system for emotion identification 
are considered. To characterize the person emotions in the model, body movements are described with linguistic variables 
and a fuzzy hypergraph for temporal events, which are transformed into the expression in a limited natural language. 
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1. INTRODUCTION 

There are a lot of modern information technologies incorporated into the human life such as Internet, 
robotics, games, video monitoring, and so on. The main purpose of these information technologies is to 
improve a human-computer interaction. But, for instance, a replacement of real persons by automated 
systems is impossible without overcoming the barrier of man-machine relationship (Orlova and Rozaliev, 
2011). The inability of machines to recognize and show emotions is one of the impeding factors. The 
development of telecommunication technologies changes the interpersonal communication. Very soon people 
will use virtual communications, which will be more effective and easy to learn but could not express 
emotions. At the same time emotions play a vital role in the human life. Emotions influence on cognitive 
processes (Bernhardt, 2010) and decision making (Petrovsky, 2009). So, it is important to recognize and 
identify the human emotions and use them. 

We developed a new approach to the identification of human emotions that is based on description and 
analysis of body movements, recognition of gestures and postures specific for the emotional states. In this 
paper, we present the methodology, models and the automated system, which are realized the suggested 
approach. 


2. IDENTIFICATION OF HUMAN BODY MOVEMENTS 

The process of identifying human emotional response is based on the idea of how the human manifests 
his/her emotions (Ilyin, 2008; Rozaliev and Zaboleeva-Zotova, 2010). 

Now various companies are actively developing automated systems for recognition, identification and 
transmission of emotional reactions. Many of these systems use web solutions based on a model SaaS 
(Software as a Service). There are also different ways for determining emotional states such as by voice, 
facial expression, body movements, physiological parameters, and so on. (Bernhardt, 2010; Coulson M, 
2004; Hadjikhani and Gelder, 2003; Laban and Ullmann,1988) 

The proposed approach to emotion identification are based on recognition and analysis of human gestures 
and poses. (Zaboleeva-Zotova et al, 2011b). 


77 



ISBN: 978-989-8533-18-0 © 2013 lADIS 


First of all, we recognize a person on video images using a technology for markerless motion capture with 
the digital video camera Microsoft Kinect. Video pictures are presented in the special animation format - the 
BVH-file, which describes poses of body skeleton and contains motion data. Such technology allows 
visualizing and analyzing different movements of person, determining areas of static or dynamic postures of 
micro and macro movements. 

To detect the borders of movements, the motor activity of person is analyzed. For the separation of 
postures, we suggest a special notion of activity, which depends on what part of body performs the 
movement. We describe the typical body movements with linguistic variables and fuzzy hypergraphs for 
temporal events, and transform these descriptions into the expressions in a limited natural language, which 
characterize the person emotions. The identification of human emotional reactions such as joy, sadness, 
anger, etc is provided by the detailed analysis of postures, gestures and motions. 
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Figure 1 . The Architecture of System for Identification of Fluman Emotional Reactions 

The architecture of computer system developed for identification of human emotional reactions is shown 
on Fig. 1 (Zaboleeva-Zotova et al, 2011a). The input of the system are moving images, sound samples and 
handwriting texts. The output of the system is information about the emotional state of the real person, which 
is expressed in a limited natural language. 


3. VECTOR MODEL OE SKELETON 

In order to define human emotional reactions by body movements, we use the vector model of skeleton, 
which is obtained from video information captured with the digital video camera Microsoft Kinect. 

Kinect camera allows obtaining three-dimensional image in all lighting conditions and without any 
requirement to the actor, who is in the frame. Data from Kinect represented as a hierarchy of nodes of the 
human skeleton. Rotation of one joint with the other, is presented in the form of quaternions (the role of the 
rotating vectors perform the bones of the skeleton) and the offset is represented as a 3 dimensional vectors in 
local to each node coordinate system. To obtain BVH-file, we have developed a new method consists of the 
following steps: 1. Getting data from the camera Kinect. 2. Determine the displacement of nodes relative to 
the parent node. 3. Record the hierarchy of key units in accordance with the specifics of the BVH-format. 4. 
Conversion of quaternions to the Euler angles. With the vector model of body movements, presented as a 
BVH-file may work most of the currently existing animation packages. 
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Vector model of the human body is the formalized representation of the movement of the person, where 
as vectors are presented bones of the human skeleton, and the angles between them correspond to the rotation 
angles of the main nodes of the human body in relation to each other. The vector model of skeleton consists 
of 22 nodes, which correspond to different anatomical joints with one, two or three axes of rotation (Fig. 2). 


Eigure 2. The Vector Model of Body Skeleton and Con'espondence between Anatomical Parts of Body and Nodes of the 

Vector Model 

Using information on structure of body skeleton presented in the vector model and motion data contained 
in BVH-files, which describe poses of skeleton, we formalize the concept of motor activity of person 
expressed in gestures as follows: 

A(At) = (Tn(At)-k,,). 

Here m is a number of time series describing movement of the body parts, TJAt) is a variation of the n-th 
time series for the time At, k„ is a coefficient that characterizes influence of the body parts on the body 
motion for the n-th time series. 

The influence coefficient can be calculated as the following sum 

~ ^i=i (Pi' ^ni) 

where i is a index of the body part, j is a number of the moving body parts, q„i is a ratio of the body part in 
the total body mass, p, is a gender coefficient of proportionality. According to biomechanical studies the 
averaged values of ratio q„i for adults are equal to 6,9% for head, 15,9% for the upper section of trunk, 2,1% 
for shoulder, 16,3% for the middle section of trunk, 1,6% fore forearm, 1 1,2% for the lower section of trunk, 
0,6% for brush, 14,2% for thigh, 4,3% for lower leg, 1,4% for foot. The gender coefficient p, is equal 
approximately to 1 for all parts of man body, and differs for various parts of woman body. 

Another important characteristic of body movement is a mobility of the joint, which is measured in 
morphology by values of the angles of flexion-extension, abduction-reduction, internal -external rotation as 
follows: 

Mjoiiit = angle (Fold+Straightening, Bringing+Abduction, In+Out). 

The maximum spine mobility is a sum of the angles of the left and right rotation around the longitudinal 
axis of the body. 

For automatic separation video districts of the individual poses and gestures, we introduce additional 
parameters, defined by the user: the minimum duration of the movement, the level of activity for poses, the 
level of activity for the movements. Next, we construct a graph of activity and find areas of the postures and 
movements. 

Poses discussed in detail in the works B. Birkenbil, G. Wilson, D. Morrison, A. Pease, were merged into 
granules, based on a similar interpretation. As it is impossible to unequivocally define the current posture 
emotional state of a person, we define the granules, which belong to the posture. 
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This allowed us to increase the reliability of a particular emotional state. Compliance granules poses and 
basic emotional states by K. Izard is shown on Fig. 3 . 
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Figure 3. Compliance Granules Poses and Basic Emotional States 


4. FORMALIZATION OF HUMAN MOVEMENTS 

In the vector model of skeleton, the movements of human body are described with the linguistic variables, 
which characterize duration of event, variation of rotation angle. The duration of event is measured in the 
frames of video image. The fuzzy temporal variable “Duration of event” includes the following set of terms: 
Do ‘Zero’, Di ‘Very short’, D2 ‘Short’, D3 ‘Moderate’, D4 ‘Long’, D5 ‘Very long’. The membership functions 
of the variable “Duration of event” are presented on Fig. 4 . 



Figure 4. The Membership Functions of the Variable “Duration of Event” 

Each group of joints with the similar values of maximal mobility is presented with the linguistic variable 
“Variation of rotation angle” that consists of the following set of terms: Bq ‘Stabilization’, B+y ‘Very slow 
increasing’, B+2 ‘Slow increasing’, B+j ‘Moderate increasing’, B+4 ‘Fast increasing’, B+s ‘Very fast 
increasing’, B_i ‘Very slow decreasing’, B_2 ‘Slow decreasing’, B_s ‘Moderate decreasing’, B^ ‘Fast 
decreasing’, B_; ‘Very fast decreasing’. The membership functions of the variable “Variation of rotation 
angle” are presented in the Fig. 5 . This linguistic variable can be adjusted on various types of the human 
movements and allow to describe, for instance, the small periodic fluctuations, such as tapping on the table, 
shaking hands or fingers, wiggle from foot to foot, and so on. 
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Figure 5. The Membership Functions of the Variable “Variation of Rotation Angle” 
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By specifying the name of the analyzed part of body, and the range of movements in the vector model of 
skeleton, one can obtain the values of rotation angles of the node relative to one of the axes X, Y or Z, which 
are stored in a separate data array. From this array there is selected a subarray, which contains the values of 
angles p, falling in the range analyzed. The angles, belonging to different frames for the same node, form a 
triangular matrix, which elements is determined by the following rule: py“p,-p/ for j>i, Pij=0 for j<i. This 
triangular matrix is used to calculate the values of the membership function of linguistic variable “Variation 
of rotation angle”. 

The movement of the joint around an axis has been described in the form of fuzzy temporal events. Since 
the events are located one after another on the time axis, the motion can be represented as a fuzzy sequential 
temporal sentence (Bernshtein et al, 2009). For example, the variation of the angle of rotation around the axis 
X for the joint “right foot” in the interval [t 4 ; ti 2 ] shown in Fig. 6 can be described as the following series of 
fuzzy temporal statements: “For the right foot there is a very slow decreasing the angle of very short 
duration. This is followed by stabilization of the angle of zero duration. This is followed by a very slow 
increasing the angle of very short duration”. 



Frame number 


Figure 6. Variation of the Angle of Rotation around the X-Axis for the Joint “Right Foot” 

The above fuzzy sequential temporal sentence can be written formally as follows: 

W= (B_i rtfDi) rtsn (Bg rtfDg) rtsn (B+i rtfD]), 

where rtf is a fuzzy temporal relationship; rtsn is a temporal relationship of the direct sequence; Bg is the 
term ‘Stabilization’, is the term ‘Very slow decreasing’, B+] is the term ‘Very slow increasing’ of the 
linguistic variable “Variation of rotation angle”; Dg is the term ‘Zero’, Dj is the term ‘Very short’ of the 
fuzzy temporal variable “Duration of event”. 


5. EVALUATION OF SIMILARITY BETWEEN THE IDENTIFIED AND 
ETALON MOVEMENTS 

In the model of fuzzy sequential temporal sentence, an adequacy of the analyzed fragment dq of a dynamic 
process and the corresponding attribute q are determined by the validity criterion J, which is represented as 
follows: 

J(q/Sq) = Fq(dq)&fiu,(Sq). 

Here Fq(Sq) is the characteristic function that establishes a semantic relationship between fuzzy values of 
the secondary attributes of a dynamic process and values of the primary attributes determining them; fiLq(^g) 
is the membership function of the term Lq of the linguistic variable L. 

The validity criterion of fuzzy sequential temporal sentence W with respect to any dynamic process S is 
written as 

J(W/S) = max,^v(J(W/S),), 
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where V is the set of all possible interpretations /. For instance, the validity criterion of fuzzy sequential 
temporal sentence W with respect to any dynamic process S for a set of fuzzy temporal events, which is 
expressed through successive attributes a, b, c, is described by the formula 

J(W/S)l ^ J(a/da)&J(b/Sb)&J(c/dc) = (Fa(dg)&^La(8a))&(Fb(dh)&^lLh(Sb))&(Fc(Sc)&^lLc(dc))- 
In our case, the analyzed dynamic process is a sequence of frames in the skeleton vector model, which 
characterizes the rotation of one of the skeleton nodes around the axis X, F or Z at a certain angle, and the 
criterion of validity is the criterion of similarity between the identified and etalon movements. So, the 
identified movements are considered as the well recognized with respect to the etalon movements if the value 
of criterion of similarity exceeds a predefined threshold. 

For example, calculate the criterion of similarity between the identified and etalon movements describing 
a rotation of the node “right ankle”. The etalon movements are presented by fuzzy temporal event, which is 
written as follows: “For the right ankle there is a very slow decreasing the angle of zero duration”. 

Let the initial data are the following time series: at the time moment to the rotation angle po=10.00 
degrees; at the time moment t 2 the rotation angle p2=6.13 degrees. So, the duration of event is equal to 2 
frames. Then by the graph of membership function pDo(8r) of the term Do ‘Zero’ of the fuzzy temporal 
variable “Duration of event” presented in Fig. 3, we find the value pjoo(8,)=0.70 for S,=t2-tg=2 frames. By the 
graph of membership function pB-i(8p) of the term Z?_/ ‘Very slow decreasing’ of the linguistic variable 
“Variation of rotation angle” presented in Fig. 4, we find the value pB-i(8p)=0.92 for 
Sp=p2-po=6.13-10.00=-3.87 degrees. Thus, the criterion of similarity J(W/S)=0.92. If the threshold is equal 
to 0.80, then the identified rotation of the node “right ankle” is similar to the etalon movements. 

6. USE FOR TEACHING CHILDREN WITH HEARING DISABILITIES 

We use information about emotional reactions to control the education of children with hearing disabilities. 
Briefly describe another developed by us system. The system is aimed for recognition and translation in real 
time gestures of the Russian language of the deaf in the text and the text in gestures. The system is intended 
for training of children with hearing disabilities and adults who need to learn sign language. It will be used in 
a test mode in school for children with limited hearing. But already now receives positive reviews. 



Figure 7. Finding Hands on the Image 
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Problem use Kinect to recognize the small gestures of hands is still unresolved, despite the successful 
application of Kinect to recognizing faces and tracking of the human body. The main reason for this low 
resolution depth map sensor. 

In sign languages in communication, information is transmitted via several channels; directly through 
hand gestures, facial expressions, lip shape, position of the body and head. Hand gestures described via hand 
position, direction of movement, shape and direction of hands. The first stage of recognition is a 
segmentation of the image received from Kinect to find the hand or both hands. Development of a method for 
finding the hand in the picture is one of the most complicated problems in the process of creating a system of 
recognition of gestures. There are several signs that can be used to detect the object on the image: the 
appearance, shape, color, distance to the subject and context. When detecting faces in the image, a good sign 
is the appearance, as the eyes, nose and mouth are always about the same proportions. Therefore, to find 
hands, we first find the face of a man, define its color and highlight the closest object. Accept his hands. Next 
we apply the developed method for finding the hands and define user gesture. An example of the recognition 
the user's hand is shown in Fig. 7. 

The system works as follows. The user enters text. The system displays an animated image of the gesture. 
A sample output of the animated gesture is shown in Fig. 8. 


r 

■ n 

■ J 


Figure 8. Animated Demonstration Gestures 
User-child repeats movement. Example showing the user gesture is shown in Fig. 9. 
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Figure 9. Show Gesture Language of the Deaf 
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The movement is recognized and checked for correctness. If not correct, the movement is shown again. If 
correct, then enter the new text. If the user starts to receive a closed posture characteristic of anger, 
resentment, it is informing the administrator and learning process can be stopped. 


7. CONCLUSION 

The identification of the human emotional states is closed to the problem of understanding what is the normal 
behavior. The variety of “normal” behavior is great. So it is difficult to draw the line between acceptable and 
unacceptable behavior. The automation of the human emotion recognition can help to solve many problems 
of relationships between people and avoid possible misunderstanding. 

Automated systems for the human emotion identification by gestures and movements can be useful and 
necessary in various areas such as communication of deaf people; education/learning; emergency services; 
monitoring unstable emotional state of pilots, drivers, operators of complex technical system, etc; monitoring 
public places to prevent illegal and extremist actions, and so on. 

In the future, we intend to use the developed approach to determine the emotional response by the 
handwritten text, and to animate human gestures and motions described in a limited natural language. 


ACKNOWLEDGEMENT 

This work is supported by the grants of Russian Foundation for Basic Research 12-07-00266-a, 12-07-00270- 
a, 13 -07-00351 -a, 13-07-00459-a, 13-07-00461 -a. 

REFERENCES 

Bernhardt D., 2010. Emotion inference from human body motion. University of Cambridge Computer Laboratory, 
Cambridge. 

Bemshtein L. et al, 2009. Models of representation of fuzzy temporal knowledge in databases of temporal series. Journal 
of Computer and Systems Sciences International, Vol. 48, No. 4, pp. 625-636. 

Coulson M., 2004. Attributing emotion to static body postures : Recognition accuracy, confusions, and viewpoint 
dependence. Nonverbal Behavior, No. 28, pp. 1 17-139. 

Hadjikhani N. and Gelder B., 2003. Seeing fearful body expressions activates the fusiform cortex and amygdale. Current 
Biology, No. 13, pp. 2201-2205. 

Ilyin E., 2008. Emotions and feelings. Piter, St. Petersburg (in Russian). 

Laban R. and Ullmann L., 1988. The Mastery of Movement. Princeton Book Company, London. 

Orlova Y. and Rozaliev V., 2011. Analysis and evaluation of the emotional reactions of the user in the articulation of 
interaction with the automated system. Open education. No 2, pp. 83-85 (in Russian) 

Petrovsky A., 2009. Theory of Decision Making. Academia, Moscow (in Russian). 

Rozaliev V. and Zaboleeva-Zotova A., 2010. Modeling the emotional state of a person based on hybrid methods. 
International Scientific-Practical Journal of Software and Systems, Vol. 90, No. 2, pp. 141-146 (in Russian) 

Zaboleeva-Zotova A. et al, 2011a. The development of the automated determination of the emotions and the possible 
scope. Open Education, Vol. 2, pp. 59-62 (in Russian) 

Zaboleeva-Zotova A. et al, 2011b. The use of fuzzy temporal statements for the description of motion in the emotional 
reactions. News Of The Volgograd State University. Series "Actual problems of management, computer engineering 
and Informatics in technical systems", Vol. 10, pp. 60-64 (in Russian) 


84 



lADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2013) 


THE ANCESTOR PROJECT: ABORIGINAL COMPUTER 
EDUCATION THROUGH STORYTELLING 


Marla Weston^ and Dianne Biin^ 

' Computer Science Department 
^Educational Support and Development 
Camosun College, Victoria, BC, Canada 


ABSTRACT 

The goal of the ANCESTOR program is to use digital storytelling as a means of promoting an interest in technology 
careers for Aboriginal learners, as well as increasing cultural literacy. A curriculum was developed and first tested with 
Aboriginal students at the LAU,WELNEW Tribal School near Victoria, British Columbia, Canada. Based on feedback 
from both teachers and students, the curriculum was updated and then tested with non-Aboriginal students. Eollowing 
more refinements, the curriculum was then delivered to Aboriginal learners using two different approaches. The first was 
a summer camp that took place within a Eirst Nations community over three weeks, and the second was a special elective 
for grade 6 Aboriginal students at Shoreline Middle School in Victoria. The results from all these deliveries highlight 
student engagement, cooperation and imagination plus an increased interest in technology. However, in spite of a rich 
Eirst Nations storytelling tradition, youth were unable to provide cultural perspective in their stories which demonstrates 
the lack of cultural transference. Next steps will include more emphasis on the integration of technology with the use of 
traditional language(s), as well as experiential components such as the inclusion of elders and time "on the land". 


KEYWORDS 

Indigenous, storytelling, literacy, culture, pedagogy, collaborative 


1. INTRODUCTION 

Within British Columbia, the Aboriginal population accounts for 4.4% of the general population, yet has a 
larger youth cohort than the general population; according to Census Canada (Statistics Canada, 2008), the 
Aboriginal youth population in British Columbia (15-24 years) will be at 54,000 by 2013. However, 
Aboriginal youth are marginalized in the education system and experience lower graduation rates from high 
school and fewer pursue higher education. To build educational participation and address colonizing 
attitudes many educational institutes in Canada are now actively looking to make curriculum relevant to 
Indigenous pedagogies and values. When asked what helped contribute to their success at the secondary and 
post-secondary level, Aboriginal learners confirmed that culturally relevant curricula, inclusion activities in 
and out of the classroom, and support systems (cultural and academic) enable an environment to succeed. 

One form of Indigenous pedagogy being incorporated into mainstream curriculum is storytelling (Burk, 
2000). An important outcome of storytelling is personal empowerment as youth incorporate traditional 
knowledge into their current learning environment rather than being passive recipients of knowledge. Telling 
or sharing a story gives value and significance to events in a student's life (Brown, 1995). The ANCESTOR 
(AborigiNal Computer Education through STORytelling) program incorporates storytelling to promote an 
interest in technology careers for Aboriginal youth and adult learners, as well as increasing cultural literacy. 
Courses and/or workshops with such a focus, along with lesson plans, encourage Aboriginal learners to build 
computer games or animated stories related to their culture and connections to the land and community. In 
an active learning environment, learners gain mathematical and computation skills, think creatively and 
reason systematically in a fun and collaborative way. By using Indigenous realities as a foundation to 
learning, the content becomes relevant to Aboriginal learners, thus creating a safe environment to build 
competencies. 
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Transferring the Aboriginal oral storytelling tradition to a digital expression has not been without 
controversy. Todd (1996) questioned whether Aboriginal world views could find a place in cyberspace. She 
argued that they are completely different ideologies. Hopkins (2006) stated that in the ten years since Todd 
expressed her views, that "Cyberspace has been occupied, transformed, appropriated, and reinvented by 
native people in ways similar to how we've always approached real space. Like video, digital technologies 
have become a medium for speaking and telling our stories." 

There are many definitions for the term "digital storytelling". It can be as simple as "using digital media 
to tell stories", to the more detailed "At its core, a digital story is a narrative expressed in digital form for a 
variety of purposes, with applications ranging from education to personal expression, record keeping to 
movement promotion and everything in between." (Sussex, 2012). Barrett (2006) feels that digital 
storytelling facilitates the convergence of four, student-centered learning strategies: student engagement, 
reflection for deep learning, project based learning, and the effective integration of technology into 
instruction. Many of these strategies fit in with the skills deemed by the "21st Century Literacy Summit" 
(2005) as key elements to developing essential digital literacy skills for the future. The Literacy Summit 
report states that "Access to tools that empower expression in these new forms must be as ubiquitous as word 
processing software or spreadsheets. In schools, tools for creating new media should be available as early as 
possible, even in primary grades, and more advanced tools provided as students progress and gain facility 
using them." (p.l4). The report encourages a community approach to literacy. 

McKeough et ak, (2008) note that "There is substantial evidence that Aboriginal youth face serious 
challenges in schooling, in general, and in literacy development, specifically." (p.l48). They emphasize the 
need to design early literacy programs that engage Aboriginal children. Since 1998, First Nation communities 
in British Columbia have participated in the Headstart Program, which builds literacy and cultural 
competencies of young First Nations children both on and off reserves. Other initiatives such as "Success By 
6" (http://www.successby6bc.ca/) are also addressing engagement and comfort with literacy for both toddlers 
and their parents. Connections to community, storytelling, culturally appropriate learning opportunities, and 
collaborative life-long learning are hallmarks of these programs. The ANCESTOR program seeks to extend 
this literacy into the realm of technology. 

The basic problems faced by Aboriginal youth in Canada are not unique. Similar issues are experienced 
by Indigenous peoples around the globe. For example in Africa, colonization policies have resulted in the 
submergence of cultural diversity through the exclusion of most African traditions from education 
(Woolman, 2001). Fleming and Southwell (2005) note that the dropout rate for Aboriginal students in the 
Australian education system is very high, and one of the key factors is the alienation of students within a 
white, Eurocentric school curriculum. Similarly, native Hawaiians are over represented in special education 
and underrepresented in higher education, which may be the result of the students' school experiences being 
different from their experiences in their home communities (Yamauchi, 2003). The premise of mono-cultural 
education is failing many Indigenous populations across the globe. It has only been in the past two decades 
that Indigenous communities and scholars have begun to deconstruct colonization policies and practices in 
mainstream education. 

The adoption of culturally relevant curricula has proved to be of value in many different countries. Eor 
example, using culturally relevant "entry points", researchers found that they could make computer science 
more relevant to students in the Kidugala Secondary School in Tanzania (Duveskog et al, 2003). Richards 
(2004) found that integrating IT education in the context of cross-cultural dialogue and interaction in Asian 
educational contexts have formed the basis for effective change. The use of technology in the Hawaiian 
Language Immersion Program (HLIP) has generally been recognized as a success, yet there remains concerns 
about the "... balance of technology {‘enehana) and traditional Hawaiian knowledge {‘ike ku ‘una)" (Yong & 
Hoffman, 2013, p.l331). This balance guides the development of curricula, as it is not only about designing 
content and learning opportunities. A culturally relevant curriculum includes the holistic learning needs of 
the learner, how the community supports the learner, and accommodates the relationship to place and culture. 


2. DESIGNING THE CURRICULUM 

Carnegie Mellon University's "Alice" is a 3D programming environment that allows learners to create 
animations for telling a story, or developing an interactive game. 
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Alice itself was developed as a teaching tool for introductory computing, and is freely available to 
download from the Alice website (http://www.alice.org). Alice is used by approximately 10 percent of U.S. 
colleges and universities, as well as in many high schools around the world (WebWire, 2007). Alice has also 
been used successfully to incorporate cultural perspectives into the teaching of programming by the 
University of Hawai'i at Hilo (Edwards et al., 2007). The library of 3D assets is dynamic and can be adapted 
to reflect culture and place, an added benefit for adapting the programming to different learning constructs. 

For Aboriginal youth, Alice has an additional advantage in that the programming environment is 
expressed in terms of a "world", which provides an effective parallel to an Indigenous world view. 
Traditional cultural expressions through storytelling and transference of history (Young-In, 2008) are done in 
a protected and respected manner to ensure relevance of place to peoples. It is this detail to creating an 
effective, interconnected world that matches the logic of the Alice environment. Thus, Alice was selected as 
the learning environment for the ANCESTOR program. The challenge was to connect the Alice world with 
an Indigenous world view, while still maintaining an effective pedagogical approach and retaining cultural 
expressions in a protected manner. An important component of the ANCESTOR program is ensuring that 
stories told within communities stays in communities. The inclusion of cultural stories within the core 
curriculum was approved to be shared as an educational tool. 

A team consisting of Camosun College faculty from Computer Science and from Aboriginal Education 
and Community Connections, plus two Computer Science students built a test curriculum. Three distinct 
curricula were created, each spanning a different time frame. These curricula included a one-day workshop 
called “Alice is fun!”, which was designed to encourage learners to create a fun, simple animation. A one- 
week workshop and a semester-long course were also developed. The curriculum was largely derived from 
the extensive online resources available for teaching Alice. In all cases, the curriculum examples had an 
Indigenous focus, using examples that are culturally relevant, respectful of traditions and knowledge, and 
appropriate to share. 


3. REFINING THE CURRICULUM 

The developed semester-long course material was first tested with grades 7 to 10 Aboriginal students at the 
LAU,WELNEW Tribal School near Victoria, B.C. Canada. As reported by Weston and Biin (2011), at the 
end of this test at the Tribal School students were: (a) more comfortable in their use of computers, (b) more 
interested in learning about computer science and programming, and (c) more interested in their cultural 
stories. In fact this third result was one of the most positive outcomes from an Indigenous cultural 
perspective. Many of the students were exposed to storyboarding and how to tell an effective story. They 
would then take this tool and seek guidance from their cultural knowledge keepers. Was this done the right 
way? Would this tell the story in the right context? What if I incorporated this dialogue or interaction? For 
one student, he further developed his story to include the SENCOTEN language (his traditional language) as 
recorded dialogue. He incorporated male and female voices into his story, using new language learners, 
which demonstrated an innate balance of form and structure in Indigenous world view. 

In spite of all the positive outcomes, there were significant problems, particularly in the way the material 
was delivered. The curriculum was generally too advanced for the students, and more repetition was needed. 
The proposed solution was to develop a series of simple video tutorials that the students could replay as often 
as needed. 

With the help of a second team of computer science students from Camosun College, a total of 17 video 
tutorials were developed and posted to YouTube (http://www.youtube.com/ancestorproject). All tutorials are 
less than 10 minutes in length, with many less than 5 minutes. All examples used in the video tutorials have 
an Indigenous theme and, in some cases, are presented as part of a larger creation story from the WSANEC 
peoples. The tutorials were built to progress from simple methods to more advanced methods, and were 
divided into beginner, intermediate and advanced levels. 

This newly revised curriculum complete with the video tutorials was tested with non-Aboriginal high 
school students as part of an access to technology course given several times at Camosun College starting in 
2011. Students taking this access course were self (or parent) selected based on their interest in technology. 
As a result, these students were very comfortable with technology and were what the 21st Century Literacy 
Summit report calls "digital natives." (2005, p.2). 


87 



ISBN: 978-989-8533-18-0 © 2013 lADIS 


Any issues these "technology savvy" students had with the curriculum were noted and immediately 
addressed. If these students were having problems, then it was likely that students from a broader background 
would have even more difficulty. 

To emphasize the storytelling approach, a formal section on storyboarding was subsequently added to the 
curriculum. The use of storyboarding also follows the recommendation of numerous authors such as Porter 
(2006) who states, "Teachers need to be diligent about requiring scripts and storyboards as a readiness ticket 
before using any technology. Scripts and storyboards ensure that the content is accurate and robust and 
demonstrate that media choices are effective and designed to support the message." (p.29). Drawing in part 
on the expertise of animation specialists from Emily Carr University (Vancouver, B.C., Canada), a new 
segment on scripting and storyboarding was added to the curriculum and set to precede any full scale 
animation. 

The revised curriculum with storyboarding was then tested with Aboriginal students. The curriculum was 
first delivered in a summer camp for the Songhees First Nation in 2012. This was followed by a fall elective 
at Shoreline Middle School (both in Victoria, BC, Canada). The audience for the summer camp and elective 
was Aboriginal youth between the ages of 11 to 13 years of age, although some mature students also 
participated on occasion. 

This revision to the curriculum supported a self-guided process for both the camp and elective 
participants in order to build comfort levels and gain digital literacy skills with the Alice programming 
environment. The offerings were then tailored to audience, time availability and resourcing needs. 

The summer camp took place over three weeks (approximately 20 hours of instruction) with participants 
between of ages of 1 1 to 22 years. With one lead facilitator, the cohort was facilitated through a cultural 
storytelling exercise where oral narrative was interpreted into a 3-D animated sequence. 

The fall elective at Shoreline Middle School ran for 7 weeks with one hour of instruction per week. 
Shoreline Middle School consists of about 300 students from grade six to grade eight with an Aboriginal 
population of about twenty-five percent. The Aboriginal community consists of students from two local 
reserves, and non-status and Metis students. An opportunity to have some students participate in this test was 
seen as a positive way of introducing digital storytelling to a small group of students as a pilot program. 

The student group at Shoreline Middle School was introduced to a scripted story within Alice to 
determine interest in pursuing the elective through the remainder of their academic term. Once ‘hooked’ the 
selected group went on to learn simple programming methods and experiment with creating their own 
characters for an animated sequence. For this group the video tutorials were not utilized; instead the lesson 
was guided by one-on-one by facilitators and educational assistants. 


4. LESSONS LEARNED 

Each of the four testing scenarios contributed significantly to our knowledge regarding learning paradigms 
for Aboriginal students. The testing scenarios and outcomes are summarized in Table 1 below. Details 
regarding the curriculum and outcomes are given in following sections. 


Table 1. Overview of the Approach for the Test Groups 


Delivery Location 

Timeframe 

Cohort 

Curriculum 

Results Summary 

1. lAu,welnew 

Tribal School 

Semester: 

1 hour per 
week 

(2010-2011) 

Computer 

science 

elective 

• Lessons 
available online 
(Moodle). 

• No video 
tutorials. 

• Students found the 
curriculum too 
advanced. 

• With only one 
hour/week students 
forgot previous 
material. 

• Needed more 
repetition. 

2. Technology 
Access Program 

3 hours per 
week for 3 
weeks 

Non- 

Aboriginal, 

selected 

• Curriculum 
revised and 
video tutorial 

• Video tutorials were 
successful, although 
some were too complex 
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(4 repetitions 
during 2011- 
2013) 

technology 

savvy 

students 

added. 

• Lessons 

available online 
(Moodle). 

and were revised. 

• Students liked moving 
at their own pace. 

3. Summer Camp 

2.5 hours 
daily 

3 days a week 
for 3 weeks 
(2012) 

Voluntary 

sign-up 

• Curriculum 
altered for time 
frame. 

• Additional 
video tutorials 
added. 

• A segment on 
storyboarding 
added. 

• Video tutorials now at 
right level. 

• Storyboarding a 
difficult concept at the 
beginning. 

• Students worked well 
together and supported 
each other. 

4. Middle School 
Elective 

1 hour weekly 
for 7 weeks 
(2012) 

Referred and 
selected 

• One-on-one 
facilitation. 

• Storyboarding 
concept 
delayed. 

• Prizes used to 
encourage 
participation. 

• One-on-one facilitation 
helped overcome the 
frustration factor. 

• Delay of storyboarding 
concept more 
successful. 

• Students very 
supportive of each 
other. 

• Students love prizes. 


4.1 LAU,WELNEW Tribal School 

Many of the students at the LAU,WELNEW Tribal School have only limited exposure to computers, and 
thus the curriculum had to find the right balance to accommodate disparate skill sets. Also, the one-hour per 
week class time meant the students forgot what they had learned the week before and had to review before 
continuing. Since the Alice environment is so rich, students would often get lost and thus frustrated. It was 
clear that more repetition was needed which led to the development of the online video tutorials. 

Even though the students had some problems with the curriculum, the teacher commented that as soon as 
the students saw the gallery elements with the cultural 3D images, they were very excited and encouraged to 
try to create traditional stories. The teacher also noted that the students taking the elective showed significant 
improvements in the analytical skills. She was very impressed by how much better the students were 
performing in their other courses such as math. 

4.2 Technology Access Program 

The "technology savvy" Access students represented a distinct contrast to the Tribal School students. Eor 
one, these students were generally older; most were grade 12 students. Eor another, all were very comfortable 
with technology as evidenced by the wealth of smart phones and tablets that traveled with the students. 

The Access students were quick to provide feedback on any part of the curriculum they felt did not meet 
their needs. As a result, topics and/or video tutorials that were too complex were quickly identified and 
corrected for the next offering. On the whole, the students were enthusiastic about the online video tutorials 
and lesson material. Students liked that they could proceed at their own pace through a lesson, replaying 
tutorials as needed. After a brief, introductory lecture at the start of each lesson, the instructor was then free 
to provide one-on-one help and encouragement. 

4.3 Summer Camp 

The summer camp had one lead facilitator, and the cohort was facilitated through a cultural storytelling 
exercise where oral narrative was interpreted into a 3-D animated sequence. 
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Once guided through a storytelling example, participants then spent their time between learning new 
programming with the self-guided tutorials available on a Moodle course site prepared for the camp and 
scripting their own one to two minute animation sequence with sounds and recorded dialogue. The 
facilitators provided problem-solving assistance when participants began building their animation sequences. 
At the end of the camp, participants screened their animations for all to see and were provided gifts for 
completing the camp. 

The video tutorials proved of great value in the summer camp. Students who were more comfortable with 
technology could work ahead at their own pace. Those less comfortable could watch the tutorial as many 
times as needed. A key component to making the tutorials successful was to keep them short and simple. In 
addition, the tutorials are of value for distance delivery as well as basic support for teachers who are not 
familiar with the Alice programming environment. 

An early segment of the summer camp was devoted to oral histories and storytelling. In this segment the 
oral tradition was reviewed and was then followed by a discussion of local, traditional stories. A series of 
books written by storytellers from the area were brought in and a traditional tale was shared with participants. 
They then had to identify a scene from within the story and create a storyboard sequence. Hand-drawn, the 
storyboard would show key camera angles, dialogue, and scene. It was difficult for youth to imagine 
sequences on their own, so the facilitator provided options for scenes and re-read certain sections of the story. 
After this exercise, participants debriefed and realized that listening to a story, building a character and 
creating a scene from the story required different skills. Participants then spent time creating their storylines 
and developing a storyboard to build their animation sequences. When programming became too advanced, 
the facilitator would build the sequence and then demonstrate back to the participant so he/she could 
incorporate into their story. 

4.4 Middle School Elective 

The fall elective was given in 2012 at Shoreline Middle School. Unlike the Summer Camp, the video 
tutorials were not utilized during the class time for this group. Since this elective was a trial to see if the 
Aboriginal students would be interested in Alice, the elective was guided one-on-one by facilitators and 
educational assistants. Another major difference when compared with the Summer Camp was related to use 
of scripting and storyboarding. For the school elective, the scripting and storyboarding segment was 
removed. Based on our experience at the summer camp, it was likely the students would find that segment 
difficult especially in such a limited time frame. To keep the students engaged, they went straight to the 
computers, and worked with them from day one to build a basic animation skill set. Carefully prepared 
animated scenarios were presented to the students, and one of the facilitators would walk through the solution 
on a projected screen while the students followed along on their own computers. Depending on the 
confidence level of the student, he or she might start trying to move ahead on his or her own. The students 
were provided with extra motivation to complete the animated sequence through small prizes being awarded 
to the first three students who completed their animation. 

The decision to not have the students in the middle school elective create a storyboard first seemed to 
work well. Walking them through an existing animated story at the beginning helped clarify the process. The 
students were then able to use this foundation, along with their imagination, to create new scenes and 
characters. As Keiran Egan states, “ [imagination is not some desirable but dispensable frill, but ... is the 
heart of any truly educational experience; it is not something belong properly to the arts, but is central to all 
effective human thinking. ... Stimulating the imagination is not an alternative educational activity to be 
argued for in competition with other claims; it is a prerequisite to making any activity educational (Egan, 
1989, p.458). 

Comparing the ease and speed at which the non-Aboriginal, “technology savvy” students in the Access 
course worked through the curriculum reveals more fundamental differences. Many Aboriginal students had 
never seen or used a flash drive. As a result, much of the first class was taken up with reviewing some basic 
computer skills. There were also literacy issues to consider when developing the curriculum. Care was 
required to not make the handouts too complex. Minimal text and the use of screen shots of the sequence and 
programming code ensured that we did not experience literacy barriers. Certainly some of the differences 
were a result of the age ranges involved, but as noted by the Canadian Council on Learning (2010), literacy 
levels of Aboriginal youth in BC are statistically lower than the general population. 
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Hence, our handout materials used different modalities of learning to ensure youth were engaged and did 
not become discouraged or frustrated into silence. 


5. CONCLUSION 

Citing numerous references and statistical results, one of the conclusions of the Canadian Council on 
Learning (2008) regarding literacy levels among Aboriginal Canadians is the need for schools to be more 
culturally inclusive of Aboriginal students and Aboriginal approaches to learning. They specifically state: "A 
number of studies have demonstrated that, in different cultures, different aspects of learning are emphasized 
and valued. For example, researchers have observed that many Aboriginal students prefer co-operative rather 
competitive learning, and that many learn through imitation, observation, and trial and error rather than direct 
instruction. Given that learning style factors can contribute to the alienation of Aboriginal students within 
classrooms, attending to these factors should contribute to more successful outcomes among Aboriginal 
students." (p.6). 

The results of our curriculum trials fully support the conclusion of the Canadian Council on Learning. 
Students in the tribal school, summer camp as well as the middle school elective worked well together and 
were keen to share new skills with each other. Walking through initial examples with the students allowed 
them to build the confidence to move forward on their own. It was important to distinguish learning the tool 
(in this case Alice), from learning how to create a story. We found that the students first needed to see what 
the tool could do before they could see how it applied to a story. As we discovered in the summer camp, 
creating or listening to a story, then building a character and creating a scene from the story required different 
skills. This was borne out at the middle school. Although the students had seen many movies, and had heard 
and read stories, it was clear that starting with creating a storyboard and script was not going to work for this 
group of students. They first needed to build a comfort level with the computer and the animation tool, before 
they were ready to let their imaginations loose. The students also gained new perspectives from their 
classmates, which led them to push their skills to new levels. 

Reviewing the results of our curriculum tests, we identified several areas upon which we can improve and 
expand. Our planned actions in these areas are as follows: 

1 . Incorporate more opportunities for Aboriginal students to work co-operatively and thus learn more 
through imitation and observation. One method of accomplishing this is to divide a traditional story among 
all members of the class. Each student, or team of students, would work on a segment of the story. The 
segments can then be assembled into a complete story at the end of the course. 

2. Take the developed curriculum to a different, less urban First Nations community. All our work thus 
far has been with Southern Vancouver Island communities. It is important to explore how different 
worldviews and story traditions will fit with our current approach. 

3. Explore using the basic curriculum to support a Language Arts program. The animation skills could 
be used in many ways such as retelling a legend while incorporating traditional language(s). The use of 
animation allows student to have a fun tool to express their learning.. 

4. Work with elders to bring more traditional stories to the students. Included as an activity in our 
planned summer camp in 2013 for the Songhees Nation, is a visit to a nearby transformation site with an 
elder. The elder will recount the tale in both English and Lkwungen (a traditional language). The students 
will then return to the classroom to create the animation in Alice. A variant of this approach has been 
successfully applied in Hawai'i (Edwards et ak, 2007). 
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ABSTRACT 

Knowledge capitalization is one of many problems facing online communities of practice (CoPs). Knowledge 
accumulated through the participation in the community must be capitalized for future reuse. Most of proposals are 
specific and focus on knowledge modeling disregarding the reuse of that knowledge. In this paper, we are particularly 
interested on CoPs of e-leaming (CoPEs) and we propose a knowledge capitalization approach within the framework of a 
CoPE, based on contextual semantic annotations. The proposed knowledge capitalization process is organized as a 
five-step cycle: 1) Acquisition and modeling, 2) storage, 3) share and reuse, 4) evaluation and 5) update. To deal with this 
process we use a contextual semantic annotation model. The annotations aim to model both CoPEs members’ tacit and 
explicit knowledge. The context represents the situation in which the members create or reuse the annotation. A prototype 
called “CoPEAnnof’ has been developed to illustrate our knowledge capitalization process. Ontological and rule-based 
context reasoning have been used to improve knowledge reuse, by adapting CoPEAnnot features according to the current 
activity context of members. 


KEYWORDS 
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1. INTRODUCTION 

Over the last two decades. Communities of practice (CoPs) have been receiving a considerable attention due 
to their potential application for managing knowledge. CoPs fulfil various functions related to knowledge 
creation and accumulation. They promote the collaboration, the interaction and the development of people’s 
kills and competencies in a particular area. CoPs of e-learning (CoPEs) represent a virtual environment to 
build, share, exchange and develop knowledge and techno-pedagogical practices among e-learning actors 
(Berkani and Chikh, 2010). 

Through their participation in the CoPE, the members create a shared repository of pedagogical 
knowledge, which includes both tacit and explicit knowledge. Therefore, they need to reuse and reap benefit 
from this repository in order to execute their activities more effectively. According to (Berkani and Chikh, 
2010), one of the main questions is how to capitalize this knowledge repository and make it more accessible 
and reusable. 

Most proposals in e-learning and CoPs are semantic -based approaches, whereby the ontological 
approaches are the commonly used. They are useful to model explicit knowledge and are widely adopted to 
index resources (Benayach, 2005) (Leblanc and Abel, 2007) (Tifous et al., 2007). However, these approaches 
are limited in modeling tacit knowledge, which requires the externalization mechanisms. Semantic annotation 
approaches are more effective for modeling both tacit and explicit knowledge, as shown in (Azouaou, 2006). 
Eor instance, the annotation of a document allows the creation of a layer of tacit knowledge. The latter is 
shared by those who reuse the document. In this case, the annotation is a way to externalize tacit knowledge. 
Otherwise, the annotation is related to the content of the document, and allows the classification and 
organization of documents. The effectiveness of semantic approaches in modeling knowledge is appreciated. 
But, the reuse of that knowledge remains under consideration. On this point, we consider that the 
preservation of the context of knowledge can overcome this issue. The context of knowledge refers to the 
parameters describing the situation in which the knowledge is modeled or reused. 
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Few studies have introduced this notion, the authors in (Azouaou, 2006) (Ouadah et ah, 2006) propose 
knowledge capitalization approaches based on semantic annotation and context. These approaches are 
specific and they are dedicated to the teacher. Their aim is to capitalize his personal knowledge and they 
don’t take into account the consideration of knowledge sharing. In addition, the notion of context has not 
been fully exploited and important aspects like context reasoning haven’t been considered. 

This work aims to propose a knowledge capitalization approach for communities of practice of e-learning. 
At first, our objective is to organize the knowledge capitalization process in the CoPE, in order to cover the 
whole life cycle of knowledge. Then, we aim to assemble the two components of semantic annotations and 
context to model CoPEs members’ tacit and explicit knowledge. At the end, our objective is to improve 
knowledge reusability by benefiting from context reasoning mechanisms such as ontological and rule based 
reasoning. 

2. CONTRIBUTION 

In this section, we describe our approach, including a process of knowledge capitalization, a contextual 
annotation model, context reasoning mechanisms and the architecture of context-aware annotation system. 

2.1 Knowledge Capitalization Process 

The knowledge capitalization process we propose is based on the knowledge capitalization model proposed 
by (Grundstein, 1995). The process is organized as a five-step cycle, where each step aims to address a range 
of co-existing issues in the three facets of Grundstein’s model: preservation, exploitation and update. 

1) Acquisition and Modeling: we propose to annotate the resources after their storage in the 
repository of CoPE. The resource is annotated based on a contextual annotation model. The latter allows 
describing the resource, externalizing tacit knowledge and representing the context of members’ activity 
during the annotation process. 

2) Storage: knowledge that represents resources and annotations are stored in a knowledge base, 
which includes a repository of resources and ontological knowledge bases of annotation and context. 

3) Share and Reuse: in this step, the ontological annotation model provides advanced reuse of 
knowledge. Reasoning capabilities on context ontology permits to adapt research, navigation and 
recommendation of annotated resources in accordance with members’ activity context. 

4) Evaluation: this step ensures the relevance and the quality of knowledge. Members may assess 
reused knowledge by means of annotations. These later allow the evaluation and enrichment of the 
knowledge base. 

5) Update: it is important to keep up-to-date the knowledge base, in order to ensure that the content 
still relevant for CoPE members. Resources and annotations can be modified or deleted when it becomes 
obsolete. The modification of an annotation is considered as the creation of a new annotation, which is 
translated to the acquisition step. 

2.2 Contextual Annotation Model 

We propose a contextual annotation model to deal with the knowledge capitalization process. The model 
represents the important aspects of annotation, which includes the description of the annotated resource, the 
representation of various elements of annotation and their links to the controlled vocabularies, as well as the 
description of members’ context during the process of creation, evaluation or reuse of annotations. The 
model is implemented using ontology. It consists of four dimensions; Resource, Annotation, Controlled 
vocabulary and Context. 

2.2.1 Resource 

This dimension represents the resource or the part of the annotated resource. It includes the following 
attributes (figure 1): 
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• URL: is the Unique Resource Identifier. 

• Title: designation distinguishing the resource. 

• Authors: creator(s) of the resource. 

• Description: represents a summary about the resource content. 

• Type: describes the type of resource (e.g. course, exercise, presentation etc.). 

2.2.2 Annotation 

This dimension represents the externalized knowledge which reflects personal knowledge of the annotator, 
and also those of recipients of annotation. Thus, those who reuse the annotation may express their judgments 
and feedback about the annotation via another annotation. This dimension is formalized based on the 
annotation models in (Azouaou, 2006) and (Mille, 2005). The conceptual model of annotation (figure 1) 
distinguishes two categories of annotation: personal and shared. 



Sub-Class ^ Data Property 


^ Sub-Property Object Property 

Figure 1 . Conceptual Model of Resource and Annotation 

1) “Personal” annotation: is associated to the author of the annotation. In the case of annotation on 
the whole resource, the annotation has the following attributes: 

• Tags: this is one or more keywords associated to the resource. It can better organize the annotated 
resources and provides also a simple and effective browsing technique. 

• Objective: represents the reason why the annotation is created. It serves to reuse the annotation, and it is 
associated with a controlled vocabulary. 

• Comment: contains free text, allowing the annotator to freely express his points of views, opinions and 
expertise about the annotated resource. 

• Reference: represents a link to another resource, this element allows the annotator to justify his opinion, 
argue or enrich his annotations. It may be a reference book, a citation, URL ... etc. 

• Expertise level: this attribute is important; people tend to trust an expert over a novice. 

• Visibility: refers to access rights to the annotation, we distinguish three types private, public and group. 

• Force: represents the value that represents the annotation for the annotator including importance and 
confidence. Based on this attribute recipients of the annotation can judge the relevance of the annotation. 

In the case of annotation on a segment of the resource, the annotation includes also the following attributes: 

• Graphical form: it represents the graphical aspect of annotation (highlighting, underlining, etc.). That is 
used to change the appearance of information to make it more visible (Mille, 2005). 

• Physical anchor describing the annotated segment in the resource. 

2) “Shared” annotation: this dimension of annotation doesn’t exist in the previous models of 
annotation. It allows members to evaluate and enhance the annotation.lt includes the following attributes : 

• Comment: a free text provided by the recipient, which allows him to express his points of view, 
interpretations, judgments about the annotation. 

• Expertise level: of the member who evaluates the annotation. 

• Score: appreciation of the value (i.e. a relevance measure) given to the annotation. 
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2.2.3 Context 

By context we mean a set of data characterizing the situation in which the member annotates, evaluates or 
reuses (view or edit) an annotation. This dimension is inspired from (Azouaou, 2006) and (Ouadah et ah, 
2006). The conceptual model of context represented in figure 2 includes two levels of context. 


Computational Context 


Machine 

Nam^ 



Figure 2. Conceptual Model of Context. 

The first level represents the generic concepts of context describing the context of annotation in general 
and it can be applied to numerous fields. It is composed of four components: 

1) Personal Context: includes the “Author” which represents the member of the CoPE, the member’s 
“Role” in the CoPE and the “Group” to which the member belongs to. 

2) Activity Context: includes the “Domain” which represents the knowledge domain (e.g. 
mathematics, physics, computer science etc.) and member’s “Activity” in the CoPE. 

3) Spatiotemporal Context: describes the following information: the “Date” and “Place” in which the 
member creates, evaluates or reuses the annotation. 

4) Computing Context: includes the “Operating system” installed on the host (Windows, Linux, etc.) 
and the “Machine” on which turns the annotation tool. 

The second level represents ontologies describing specific concepts of context. The ontology of CoPE 
(Berkani and Chikh, 2009) describes the concepts related to CoPE, such as: member, role, activity. ACM 
Computer Classification System (ACM CCS) (ACM, 2012) is used to describe computer science domain. 
But, other ontologies of location and time for instance can be also considered. 

2.2.4 Controlled Vocabulary 

This dimension represents the ontologies associated with the different elements of annotation like tags and 
attributes (e.g. graphical form, objective of annotation, etc.). We opt for the ontology proposed in (Mille, 
2005), which presents a rather comprehensive list of annotation graphical forms. As far as vocabulary 
associated to the objective of annotation, we reuse the ontologies proposed in (Azouaou, 2006), describing 
teachers’ annotation objectives. Thereafter, other controlled vocabularies can be developed. 
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2.3 Context Reasoning 


Formal approaches for context modeling such as ontology, offer many advantages. The foremost advantage is 
the inference capabilities. Context Reasoning aims to check consistency of the model as well as to infer new 
information about context and to derive high level of context. Indeed, the contextual information provided by 
the environment (system, user, sensors, etc.) leads to elementary data about context, whereas some contextual 
information is useful only if it is combined with other elementary or composite contexts. The reasoning tasks 
in this work are grouped into two categories, ontological and rule based reasoning. 

1) Ontological reasoning: context ontology is represented using OWL-DL language. The standard 
reasoning rules supported by this language are, in particular, subClasseOf, subPropertyOf, 
TransitiveProperty, disjointWith, inverseOf, etc. 


Ontological reasoning rules 

subClassOf : { ?A rdf s : subClassOf rdf s : subClassOf ?C) -> { ?A rdf s : subClassOf ?C) 

disjointWith: { ?A owl : dis j ointWith ?B),{?X rdf: type ?A),{?Y rdf: type ?B) -> 

{ ?X owl : dif ferentFrom ?Y) 

inverseOf :{ ?P owl : inverseOf IQ), (?X ?P ?Y) -> (?Y ?Q ?X) 

Explicit context 

Implicit context 

<owl:Class rdf: ID="ActivityContext"> 

<owl:Class rdf : ID="Activity"> 

<rdf s : subClassOf > 

<rdfs : subClassOf> 

<owl:Class rdf : ID="Context"/> 

<owl:Class rdf : ID="Context"/> 

</rdfs : subClassOf> 

</rdf s : subClassOf > 

</owl : Class> 

</owl : Class> 

<owl:Class rdf : ID="Activity"> 

<Conception rdf: I D=" ScenarioConception "> 

<rdf s : subClassOf > 

<Analyze rdf : ID="ScenarioConception"> 

<owl:Class rdf: ID=" Activity Cont ext " /> 

Error 

</rdfs : subClassOf> 

<Group rdf : ID="Groupl"> 

</owl : Class> 

< Contains rdf : resource= "# Author 1 " /> 

<owl:Class rdf : ID="Analyze"> 

<owl : dis j ointWith> 

<owl:Class rdf: ID="Conception"/> 

</owl : dis j ointWith> 

</owl : Class> 

<owl : Ob j ect Property rdf : I D= "Belongs "> 

<owl : inverseOf > 

<owl : Ob j ect Property rdf : I D= "Contains " /> 
</ owl : inverseOf> 

<rdf s : range rdf : resource="#Group"/> 

<rdf s : domain rdf : r esour ce="# Author "/> 

</owl :Obj ectProperty> 

<Author rdf : ID="Authorl "> 

<Be longs rdf : resource="#Groupl "/> 

</Author> 

</Group> 


Eigure 3. Ontological Reasoning. 

The figure 3 shows a part of ontological reasoning rules represented in first order logic, this with some 
examples illustrating the use of these rules. According to the context ontology, we can define the concept 
“Activity” as subclass of the concept “Context” using “subClassOf’ rule. Furthermore, we can use 
ontological reasoning via the rule “disjointWith” to infer a contradiction when the instance 
“ScenarioConception” is defined as instance of both classes at the same time. Also, a new context that 
“Groupl” “Contains” “Authorl” can be implicitly deduced based on “inverseOf’ rule. 

2) Rule-based reasoning: some contextual information cannot be easily inferred using ontological 
reasoning. Accordingly, we propose to use a flexible reasoning mechanism based on predefined rules. These 
latter are described with Generic Rule Language specified by Jena API and based on first order logic, aiming 
to deal with the third step in the knowledge capitalization process. It allows deducing additional information 
about the current context of members and consequently adapting the reuse of annotated resources. 


Table 1. Context Tuples 


ID_Context 

ID_Author 

ID_Group 

Role_Name 

Activity_Name 

Domain_Name 

Cl 

Authorl 

Groupl 

Manager 

Conception 

E-leaming 

C2 

Author2 

Group2 

Coordinator 

Conception 

E-leaming 

C3 

Author3 

Groupl 

Moderator 

Conception 

Distance Learning 


97 




ISBN: 978-989-8533-18-0 © 2013 lADIS 


The tuples in table 1 correspond to individuals of Context. The first tuple represents the current context of 
annotation “Cl”. Context reasoning basis on the other context tuples and the rule “R5” (Figure 4) infers a 
new context that the context “Cl” has the same group and the same activity as the context “C3”. More 
precisely, the rule R5 defines the relationship “SameGAc” between two instances of “context” concept, when 
their authors belong to the same group and execute the same activity. This rule is based on the relationships 
defined in the other inference rules (“Sameidc”, “InC”, “SamePerson”, “SameGroup” and “SameActivity”). 

[Rl: (?cl prefix : ID_Context ?id_cl) (?c2 prefix : ID_Context ?id_c2 ) equal {? id_cl , ?id_c2 ) -> (?cl 

prefix : Sameidc ?c2)] 

[R2: (?c rdf:type prefix : Context) (?a rdf:type prefix : Author) (?c prefix : Sameidc ?a)-> (?a prefix:InC 

?c) ] 

[R2: (?al rdf:type prefix: Author) (?a2 rdf:type prefix: Author) (?al prefix : ID_Author ?idal) (?a2 

prefix : ID_Author ?ida2) equal { ?idal, ?ida2) -> (?al prefix : SamePerson ?a2) ] 

[R3: (?gl rdf:type prefix: Group) (?g2 rdf:type prefix : Group) (?gl prefix : ID_Group ?idgl) (?g2 

prefix : ID_Group ?idg2) equal { ?idgl, ?idg2) -> {?gl prefix: SameGroup ?g2) ] 

[R4: (?acl rdf:type prefix : Activity) {?ac2 rdf:type prefix: Activity) (?acl prefix: activity ?idacl) (?ac2 

prefix : activity ?idac2 ) equal {? idacl , ?idac2 ) -> (?acl prefix : SameActivity ?ac2)] 

[R5: {?cl rdf:type prefix:Context) (?c2 rdf:type prefix: Context) noValue { ?cl prefix : Sameidc ?c2) (?al 

rdf:type prefix: Author) (?a2 rdf:type prefix: Author) (?al prefix:InC ?cl) (?a2 prefix:InC 

?c2 ) noValue { ?al prefix : SamePerson ?a2) (?gl rdf:type prefix : Group) {?g2 rdf:type prefix: Group) {?al 

prefix : Belongs ?gl) (?a2 prefix : Belongs ?g2) (?gl prefix : SameGroup ?g2) (?acl rdf:type 

prefix : Activity) {?ac2 rdf:type prefix: Activity) {?al prefix:Executes ?acl) {?a2 prefix:Executes 

?ac2) (?acl prefix: SameActivity ?ac2) -> (?cl prefix: SameGAc ?c2) ] 


Figure 4. Reasoning Rules 


2.4 Context-aware Architecture for CoPEAnnot 

Here, we propose a context-aware architecture for our annotation system called CoPEAnnot. Many 
researchers have proposed several context-aware architectures and most of them are proposed in pervasive 
and mobile computing domain. The works in (Azouaou and Desmoulins, 2006) and (Ouadah et al., 2009) 
proposed architectures for context-aware annotation systems. As most available architectures, they don’t 
permit context reasoning and inference. The latter becomes a vital requirement for context-aware systems in 
order to facilitate the adaptation task. Our architecture (see figure 5) differs from the previous ones at the 
reasoning support that provides. It consists of two main components: context management and annotation 
management. This separation is inspired from (Chaari and Laforest, 2005). The authors suggest that the body 
of application must be designed in isolation from contextual data. 

Annotation interface 



Figure 5. CoPEAnnot Architecture. 
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1) Context management: is responsible for context acquisition, reasoning and adaptation; 

a) Context acquisition: this module is responsible for collecting contextual information from different 
sources (operating system, learning environment, user, user model, physical sensors, etc.), for interpreting 
contextual information (transform them into more useful and meaningful contextual information) and for 
their storage in accordance to the ontological model of context. 

b) Reasoning engine: is the brain of our architecture. It is in charge of reasoning about contextual 
information acquired by the acquisition module. Based on ontological reasoning and rule-based reasoning, 
the reasoning engine infers information about annotations’ context which is semantically closest to the 
current context of annotation. 

c) Adaptation module: This module adapts the functionalities of context-aware system according to 
contextual information provided by the reasoning engine. 

2) Annotation management: is in charge to manage annotations, it includes the following major 
steps: 

a) Annotation management module: is in charge to insert, store and update, research, navigation and 
recommendation of annotations. These last three features are adapted according to the current context of the 
annotation. 

b) Annotation interface: represents the graphical interface that allows the exchange and the interaction 
between the CoPE members and the annotation tool. 

The context knowledge base (CKB) contains the OWL ontology defining the context. It contains also a 
set of inference rules which is processed by the reasoning engine. The annotation knowledge base (AKB) 
includes the repository of resources and the OWL ontology defining the annotation model and their 
controlled vocabularies. 


3. IMPLEMENTATION 


A prototype system has been developed based on the above architecture and annotation model. Here we give 
a brief description about the implementation of CoPEAnnot and their functionalities. Ontological models are 
developed using Protege ontology editor. The system is implemented in client-server architecture. The client 
is the user who has as browser Mozilla Lirefox, in which the annotation tool constitutes a browser extension 
(plug-in). Graphical interface was built using XUL, DOM, JavaScrip and CSS. AJAX technology is used to 
insure the communication between the client and the server which is sort of http request/response. On the 
server side, we used Tomcat as a Servlet container as well as a web server. Servlet are java programs that 
used to handle http requests/responses. Jena frame work provides a programming environment for RDL, 
RDLS, OWL and SPARQL and it also includes several internal reasoners, where the rules engine supports 
generic inference based rules. 
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Eigure 6. Screenshots of CoPEAnnot. 
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The annotation extension is constitutes of tow toolbars, the first one provides the following main features: 
CoPEAnnot (Home), Resource, Annotation, Navigate, Search and Help. The second one provides a graphical 
form palette. Home sidebar shows the tag cloud and the recommended annotations adapted to the current 
context of the member (see figure 6 part B). Members can annotate (see figure 6 part A) any resource already 
exists in the knowledge base. The tool enables annotation of different types of resources and the annotation 
on segment is considered for only html resources. Thereafter, members can also edit, share, and evaluate 
annotation. In addition to the standard features of navigation, the tag cloud facilitates access to knowledge 
and it enables faster discovering of knowledge. The tool provides also contextual semantic search based on 
controlled vocabularies (see figure 6 part C). 


4. CONCLUSION 

In this paper, we have investigated how context-based semantic annotations can be used in a knowledge 
capitalization process, dedicated to a community of practice of e-learning. We have proposed an approach 
which first aims to organize the knowledge capitalization process in the CoPE, and to cover the whole life 
cycle of knowledge. Then, the proposed contextual annotation model serves for modeling tacit and explicit 
knowledge of CoPE, and it favour knowledge sharing. Ontological and rule-based reasoning represent the 
brain of the proposed context-ware architecture of CoPEAnnot. They have been used to adapt the annotation 
tool features according to the current activity context of members. Our future work will focus on the tool 
validation. To achieve this goal, some experiments are to be performed within a community of learners. We 
also plan to improve CoPEAnnot by extending reasoning capabilities on other elements of context and 
developing controlled vocabularies. 
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ABSTRACT 

This study presents a conceptual framework for supporting mobile peer assessment by incorporating augmented reality 
technology to eliminate limitation of reviewing and assessing. According to the characteristics of mobile technology and 
augmented reality, students’ work can be shown in various ways by considering the locations and situations. This study 
proposes a novel mobile peer-assessment system which incorporates augmented reality into the reviewing and assessing 
processes. The mechanism enables students to enhance work interpretation, frequently interact with peers, represent their 
thinking and reflect upon their own works. Moreover, the mobile AR technique provides personalized and location-based 
adaptive contents that enable individual students to interact with the mixed reality environment and to observe how works 
are possibly applied to the real world in the future. The overall process facilitates students in reviewing works based on 
various dimensions, acquiring important knowledge, fostering critical thinking skills and reflection as well as promoting 
meaningful learning. 


KEYWORDS 

Augmented reality, peer assessment, mobile learning 


1. INTRODUCTION 

Peer assessment has become increasingly popular in education due to the support of group learning and the 
enhancement of learning effectiveness. In peer assessment process, students participate cognitive activities 
including doing assignments, devising assessment criteria, reviewing, summarizing, clarifying, providing 
feedback, diagnosing errors, identifying missing knowledge or deviations and evaluating the quality of peers’ 
work (Van Lehn, et al., 1995; Liu, et al., 1999; Sitthiworachart & Joy, 2003). The majority of previous studies 
emphasize conditions, methods and outcomes of peer assessment and focus on the quality of students’ work, 
domain-specific skill and peer assessment skill for outcomes (Van Zundert, et al., 2010). In recent years, 
mobile technology provides the potential of creating innovation learning experiences. Students can acquire 
learning materials, share ideas, and construct knowledge anytime anywhere by using their own handheld 
devices. In order to eliminate the limitation of space and time, mobile peer assessment positively influences the 
assessment methods and outcomes and enables students to submit their own work, review peers’ work, mark 
and provide feedback conveniently. 

However, during the peer assessment process, providing students with sufficient information to review 
peers’ work is preferable. Augmented reality (AR) is the technology that provides the right contents at the right 
place at the right time. The mobile AR technique is able to overlay virtual objects on the real work to present 
rich information to students and construct meaningful presentation by combining location-awareness and 
contextual learning. There is a positive relation between providing students with the opportunity to review 
peers’ work based on the mobile AR technique and reflecting upon their own work. 

In order to provide full insight into effective peer assessment processes, issues regarding content 
presentation as well as assessment methods require more attention. 
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Therefore, this study presents a conceptual framework for providing intelligent and mobile supports 
through incorporating the AR technique to enhance work presentation and the effectiveness of peer assessment. 
In this framework, students are able to review peers' work by using various dimensions and receive assessment 
results immediately. The difficulty of reviewing peers' work and understanding peers' thinking can be resolved 
and sufficient information representation enables accurate assessment. Most importantly, appropriate 
assessment criteria and rich feedback facilitate students to reflect upon their own work and improve the quality 
of their work. 


2. PEER ASSESSMENT IN MOBILE LEARNING 

Peer assessment has been widely recognized as an educational arrangement in which students assess peers’ 
work and provide feedback (Van den Berg, et ak, 2006), as well as a learning tool for improving student’s 
performance in collaborative learning environment (Topping, et ak, 2000). Various studies related to 
education, business, health and science on self and peer assessment in higher education have been proposed 
(Searby Ewers, 1997; Ballantyne, et ak, 2002; Prins, et ak, 2005; Price & O’Donovan, 2006). These studies 
reveal that students who involve in the interactive assessment process can enhance their interpretation and 
reflection. Regarding how to effectively involve students in peer assessment, these processes including 
exploration of assessment criteria, presentation of works, assessment methods, coordination of assessment and 
feedback are very critical (Chen, 2010 ; Lan, et ak, 2012). Most studies focus on the conditions, methods and 
outcomes (Van Zundert, et ak, 2010) and have proposed computerized-based peer assessment systems to 
support the assessment process (Davies, 2000; Lin, et ak, 2001). Appropriate technology applied in peer 
assessment can assist the reviewing and assessing activities. Computer networks facilitate students to 
participate in assessment activities anytime anywhere and enable teachers to review assessment progress. 
On-line peer assessment systems that can do away with conditions restricting various assessment activities in 
classrooms can eliminate the time and the cost in communicating with each other and printing out student work 
or assessment forms. 

In recent years, students attempt to learn in various locations, and therefore mobile learning is becoming 
widespread. Mobile technology provides the potential of creating innovative learning experiences that can take 
place anytime and anywhere (Shih, 2010). Because of the characteristics of mobile technology such as 
ubiquity, smaller size, comparative affordability, and the prevalence of wireless networks, more and more 
researchers have developed application on handheld devices such as mobile phones, tablet computers and 
PDAs to support learning activities. Some studies have proposed the critical issue of how to use handheld 
devices to enhance assessment (Penuel, et ak, 2007; Shin, et ak, 2007). Students can use handheld devices to 
flexibly conduct project-based learning and self-assessment inside and outside classrooms. A few researchers 
have reported the findings about how to use mobile technology for self- and peer-assessment (Chen, 2010). 
Chen indicated that combining mobile technology with the concept of round-table presentations, the mobile 
self- and peer-assessment system can assist teachers to arrange assessment activities more flexibly and make 
students more attentive to presentation, interaction and feedback in the assessment process. However, most of 
these studies emphasize the exploration of assessment criteria, marking process and the promotion of feedback 
to enhance the effectiveness and reflection of self- and peer assessment. Actually, it is a very critical issue that 
students’ work can be presented in detail during the assessing process. Through reviewing peers’ work, 
students can understand how to mark and reflect upon their own work. 

According to the characteristics of mobile technology, students’ work can be shown in various ways by 
considering the locations and situations; moreover students can communicate with peers as well as observe 
peers’ work anytime anywhere. This study proposes a novel mobile peer-assessment system which 
incorporates augmented reality into the reviewing and assessing process. The mechanism enables students to 
enhance work interpretation, frequently interact with peers, represent their thinking and reflect upon their 
work. Through the reviewing and interactive process, assessment accuracy and quality can be improved. The 
overall process facilitates students in fostering critical thinking skills and reflection as well as promoting 
meaningful learning. 
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3. PEER ASSESSMENT WITH MOBILE AUGMENTED REALITY 

Augmented reality (AR) is the technology that shows the right contents for the right device to the proper person 
at the right place and at the right time (Chang & Tan, 2010; Chan, et ah, 2010). It can overlay virtual objects on 
the real world to fulfill the feeling of immersion and therefore supplements user’s everyday life with 
information, images, sounds, and other sensory information from their device. Shortly to say, through putting a 
virtual layer of information over the real world, AR pretends that virtual objects are real and presented at the 
right place. The widely accepted definition of AR is “Augmented Reality allows the user to see the real world, 
with virtual objects superimposed upon or composited with the real world. AR supplements reality, rather than 
completely replacing it.”(Azuma, 1997; Azuma, et ah, 2001). AR is thought to present certain advantages over 
more traditional ways of accessing information (Anastassova, 2007). 

Alongside mobility, development of positioning technologies make it possible to keep track of students 
and provide them with tailored learning contents based on their real-time locations. Furthermore, 
location-based e-learning provides a personalized learning experience and helps in keeping the students 
engaged in the learning activities and enhancing their effectiveness (Chen, et ah, 2007). Previous studies have 
indicated that the combination of location-awareness and a contextual learning approach can enable students to 
better construct meaningful contextualization of concepts (Michie, 1998; Patten, et ah, 2006). For the purpose 
of locating virtual information at the right place in real word, tags or markers are necessary for recognition. AR 
recognizes the tag and gets its position as the position of the corresponded virtual information. There are two 
types of tags: one is so called “AR ToolKit marker”. It’s a monochrome graph surrounded by a square frame. 
The other one is full-on image recognition. The square frame of “AR ToolKit marker” transforms to a 
parallelogram when it be projected on the screen. By reversing this procedure of mapping a parallelogram to a 
square, the position and direction of the square frame in the real world can be detected, and then the virtual 
object information can be overlaid on the screen of the handheld device. However, real objects can be observed 
in various dimensions, but it is more difficult to recognize real objects than “AR ToolKit marker”. Currently, it 
is still not quite at the stage of full-on image recognition, but other researchers are working on it. Because real 
object recognition does not need extra tags, it would become the most popular approach in the near future. 

Few years ago, if someone wanted to show virtual information on real objects, he needed to wear some 
machines on his neck. These machines include one camera to capture images of the real word, a wireless 
network transporter to send images to computer systems and get feedback information that merges virtual 
objects with real-world images seamlessly, and a projector to show virtual information on real objects. 
However, it is not convenient at all. In recent years, significant advancements related to wireless and mobile 
technologies make handheld devices which combine several utilities to be the most convenient platform for the 
AR technique. The camera on the handheld device can capture images of real world, a compass can detect the 
direction of user’s face, the GPS receiver can locate the position of users, and the monitor can show the images 
of the outcome that the mobile AR technique create, which can be text, table, image, video and their 
combination. Even more, extra components such as buttons or tables can be included for interaction. The 
mobile AR technique provides pliable mobility and a location independent service without constraining the 
individual to a specific area. According to the NMC Horizon Report 2012 K-12 Edition, AR supports visual 
and highly interactive forms of learning in education. Students can use it to construct new understanding based 
on interactions with virtual objects that bring underlying data to life as it responds to user input (NMC Horizon 
Report, 2012). Numerous researches has proposed that the AR technique can help students to learn in serious 
games, language learning, e-books, storytelling, driving guidance, and so on (Azuma, 1997; Van, et ah, 2010; 
Chen (& Tsai, 2010). By this way, AR holds the possibility to revolutionize the way in which information is 
demonstrated to people and has great potential for on-demand, context-aware, and collaborative training 
(Hollerer, 2001). Moreover, the mobile AR technique provides personalized and location-based adaptive 
contents for individual students to interact with the mobile viewing environment, and see how works are 
applied to the real environment in the future at the current place. 

According to the above mentioned, the mobile AR technique can obviously support students to review 
peers’ work during the peer assessment process. Eormerly, in a design course, students only review the work 
based on assessment criteria such as originality, produce skill, colour scheme and so on but cannot view the 
usability of the work in the future in this environment where the assessor located. The most important 
functionality of a location-based mobile AR technique is to provide the proper contents according to students’ 
current location. 
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The relevant applied contents in students’ vicinity will be presented by the mobile AR technique 
automatically while students walk in the area. For example, how a painting can be hanged on the wall or 
become a fresco or how a handiwork will be if it is rebuild to a sculpture putting in this environment. "The 
incorporation of various rich sensors into new phones such as GPS location, wireless sensitivity, compass 
direction, accelerometer movement as well as sound and image recognition is enabling new ways in which we 
are able to interact with the world around us." (Nokia Research Center, 2009). The mobile AR technique can 
fuse digital media with the physical world to create the proper conditions for locative, contextual and 
situation-based demo scenarios. In this study, during the peer assessment process, assessors not only assess the 
works presented in front of them but also view the future application of target works. Therefore, assessors can 
judge the design skill of designers as well as the usability of the work in the future. 


4. SYSTEM REALIZATION AND ILLUSTRATIVE EXAMPLE 


4.1 System Architecture 

Augmented reality is defined as a real-world environment whose elements are built upon computer -generated 
sensory input such as sound, video, graphics or GPS data. In the educational field, there are many situations 
cannot be experienced in the classroom. Augmented reality is the latest technology that can accommodate or 
modify their learning experience to their specific needs. So what AR allows us to do is to see virtual objects in 
a real world environment with the aid of camera and some display devices (monitor or head mounted display). 
Subsequently, the procedure of peer assessment enriched by the mobile AR technique is described. This 
procedure shows how AR can enhance the effectiveness of reviewing and assessing during peer assessment. 
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Displays 
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Figure 1 . The Architecture of the Mobile AR Technique 
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The architecture of the mobile AR technique includes three parts such as hardware repository, persistent 
storage, and works demonstration as shown in Figure 1 . In hardware repository, 2d/3d unique tags which are 
able to recognize and present virtual objects in right locations are necessary and mobile displays are included. 
Students’ works and expositions of design about these works are categorized into persistent storage. These 
materials provide extra information of virtual objects. In the part of work demonstration, the result that overlay 
virtual object images on real object images can be presented on handheld devices such as mobile phones or 
tablets. Through incorporating the mobile AR technique into peer assessment. Figure 2 shows the framework 
of the mobile-AR peer-assessment system (MARFAS). 



Figure 2. The Framework Of The Mobile Augmented Reality Peer Assessment System 

There are three databases in the cloud, including the student profiles, the AR and virtual object database and 
the assessment database in Figure 2. At the beginning of the assessment activities, students must login to 
authenticate their identities. All data related to students have been built in the user database. Subsequently, the 
target work shows up in front of assessors, the system goes on getting all information including assessors’ 
location, the direction that they face and the situation such as indoor or outdoor. These local data is collected by 
handheld devices and sent to the system. During the peer assessment process, all procedures are parted into 
three modules including the authentication module, the context aware module, and the AR interactive module. 
The authentication module enables right people to get right information to assess right works. The context 
aware module enables assessors to use right device to receive right context for assessment, and the AR 
interactive module enables assessors to review peers’ works conveniently and intuitively such that the 
assessment can be more diversified and every assessee learns more from other works. In the context aware 
module, the system judges these local data and then choices a proper context for the assessor from the virtual 
object database. All data are ready for the AR technique to overlay on the real world image, and thus assessors 
mark these works more conveniently and accurately. 
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4.2 Walk-through Illustrative Example 

Peer assessment was facilitated by working in small groups of three to four students. These students were better 
able to compare feedback from different peers to determine its relevance (Van Zundert, et ak, 2010). Therefore, 
in this study, 50 undergraduates major in visual communication design enroll a fundamental design course and 
are arranged in small groups. The teacher assigns a painting work and then students can receive the notification 
on their own mobile phone or tablet. Students have three weeks to prepare their drafts and the exposition of 
their design ideas after which these are uploaded onto MARPAS, a mobile augmented reality peer assessment 
system. The AR application constructs the relation between the image of the draft and the exposition. 
Subsequently, during the assessment process, the teacher designs two activities in which students have to mark 
peers’ work in an indoor environment as well as in an outdoor environment. These drafts are printed out as tags 
and posted in an exhibition and on an outside wall. The assessment situations in indoor and outdoor 
environments and the interfaces of MARPAS are presented in Figure 3. 


In the indoor situation, assessors go to the exhibition to capture the tag through their own camera on 
handheld devices. Then, they can see the work can be constructed as a physical produce in the real world. The 
introduction of assessees’ work and the assessment criteria can be shown on the device at the same time. In the 
outdoor situation, assessors go to outside to capture the tag on the wall, and then the visual work is presented on 
the wall in the real world. Assessors can review the assessees’ information and assessment criteria as well. 
However, assessment criteria are different based on the varied situations. For example, assessment criteria 
including suitable, originality and colour scheme have to be considered in the indoor assessment, and assessors 
mark the dimensions of suitable, exquisite level and usability in the outdoor assessment according to the 
features of outdoor situation. By this way assessors can mark the usability of the work at the current location. In 
addition, all assessment related to the work can be presented simultaneously on the device as shown as Figure 
4, and therefore assessors are able to review other assessors’ assessment and assessees can receive the 
assessment results. 



Figure 3. The Demonstration of the MARPAS 
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Eigure 4. The Representation of Assessment Results 

According to the different surroundings, students are not only able to acquire the relative explanation and 
representation of work but also apply appropriate assessment criteria that produce sufficient assessment results 
to mark peers’ work. MARPAS facilitates students to observe other assessors’ marking as well as receive 
assessment feedback. Therefore, students can reflect upon their work according to the various and meaningful 
feedback received. 


5. CONCLUSIONS 

This study has presented a framework for providing intelligent and mobile supports to enrich peer assessment. 
The limitation of time, space and devices can be eliminated. In this framework, students can review and assess 
peers' work represented with AR technology through combining virtual objects with the real world. Mobile AR 
technology provides flexible mobility and location-based adaptive contents to interact with the assessing work 
and the real world for individual students. Students can bring their own handheld devices to capture and acquire 
appropriate information at the right time in the right situation. By incorporating the techniques of AR, the 
proposed framework enables students to review peers' work in various ways and students can receive the 
assessment results immediately. The difficulty of reviewing peers' work and understanding peers' thinking can 
be resolved and sufficient information representation enables accurate assessment. In addition, appropriate 
assessment criteria and rich feedback facilitate students to reflect upon their own work and improve the quality 
of their work. Although the proposed framework has indicated the assistance of incorporating AR in peer 
assessment, considerable work remains to be done, including further large-scale classroom experiments and 
system adaptability. 
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ABSTRACT 

Research into virtual role-based learning has progressed over the past decade. Modem issues include gauging the 
difficulty of designing a goal system capable of meeting the requirements of students with different knowledge levels, 
and the reasonability and possibility of taking advantage of the well-designed formula and techniques served in other 
research fields to improve role-based tutoring. In this paper, we attempt to develop a comprehensive and adaptable goal 
system and intelligent tutors in an educational bioscience game by proposing a hybrid approach. Our solution supports 
multi-user collaborations and competitive play, and integrates a data mining model to help discover student play patterns. 
The overarching aim is to make informed tutoring decisions, and improve student learning and efficiency, as they work 
through each module in the game. 


KEYWORDS 

Collaborative Learning, Exploratory Technologies, Student-Centered Learning 


1. INTRODUCTION 

Immersive virtual environments (IVEs) offer an excellent opportunity for science education. However, the 
tutoring designed in these games typically considers only the students’ current actions, overlooking past 
activities where important play patterns may be hidden. In the real world, a good instructor teaches different 
students with different strategies, and thus, different advice might be given based on the severity of the 
mistakes made. For example, a lucky guess should be remediated when the correct answer is given by chance 
without reasoning and analysis. This work proposes to classify past mistakes by integrating data mining 
techniques to analyze student play history, in order to uncover important play patterns to create focused, 
individualized tutoring strategies for students. 

1.1 Context 

There have been a number of serious games developed using role-based learning, such as the Geology 
Explorer (Saini-Eidukat et ah, 1998) for geosciences and earth science education, the Visual Program (Juell 
1999) for AI education, the ProgrammingLand MOO (Hill and Slator, 1998) for computer science education, 
and the On-A-Slant village (Hokanson et ah, 2008; Slator et ah, 2001) for anthropology education. 

1.2 Intelligent Tutoring 

The intelligent tutors discussed in this work are deployed in the WoWiWe Instruction Co version of the 
Virtual Cell, a virtual, multi-user space where students fly around a 3D world and practice being cell 
biologists in a role-based, goal-oriented environment (Borchert et ah, 2013; White, et ah, 1999). 


109 



ISBN: 978-989-8533-18-0 © 2013 lADIS 


It focuses on providing an authentic problem-solving experience that engages students in the active 
learning of the structure and function of the biological cell (Slator et ah, 2006). Three modules have been 
completed: the Organelle Identification module populated with sub-cellular components where students are 
required to identify the nucleus, endoplasmic reticulum, Golgi apparatus, and so forth using deductive 
scientific approaches (reasoning, analysis, assay etc.); the Electron Transport Chain (ETC) module which 
demonstrates the respiration process and requires the student to understand the movement of hydrogen and 
electrons when ADP is converted to ATP in the mitochondria; and the Photosynthesis module which teaches 
students the process of photosynthesis by asking them to repair damaged photosynthesis reactions in the 
chloroplast. In each module of the Virtual Cell game, a guide is on hand to direct them to their next task, and 
a tutor is available when students struggle in accomplishing their learning goals. 

1.3 Project Overview 

This paper can be summarized as follows. Eirst, we design a comprehensive goal system that is adaptable to 
students with different knowledge levels in a role-based, goal-oriented immersive virtual environment (IVE). 
The learners are assigned specific tasks in accordance with their learning goals covering various components 
and organelles of a cell in the first module. The difficulty of the goals increases progressively as the student 
works through each task, and the level can be adapted to meet the requirements of educating different 
students such as high school students and college undergraduates. 

Second, to provide more individualized tutoring to students who have difficulties in accomplishing their 
learning goals, we propose a data mining model to analyze student play history, aiming to discover non- 
obvious but important patterns to help make better tutoring decisions. Eor example, two kinds of tutoring are 
provided based on the specific type of mistake made; blind tutoring is initiated if the discovered patterns 
show the mistake was made by chance, while oriented tutoring is undertaken if the uncovered patterns 
implied the student may have fundamental confusion between two organelles. 

We also developed a library of problem-oriented knowledge to help locate the confusions that students 
may have as they explore in the Virtual Cell. Supported by this library, tutors act as sub-topic experts, keep 
an eye on student progress, and match the current student’s actions to previous students’ actions, allowing the 
tutor to ask relevant questions that may be preventing the student from moving forward. 

Last, we employ ontology mapping techniques to improve the data quality of the student play history and 
model student learning activities. Eor example, tutoring decisions depend to a high degree on the type of 
diagnostic assay the student is performing. Therefore, the agents developed in this work are capable of 
offering more individualized and problem-oriented tutoring. 


2. THE GOAL SYSTEM 

There are three modules in the Virtual Cell: the Organelle Identification (ID) module, the Electron Transport 
Chain (ETC) module and the Photosynthesis module. They are designed to help students improve scientific 
reasoning and their understanding of the scientific method (Borchert et ah, 2010). 

2.1 ID Module Goal System 

The ID module was developed to provide an introduction to game play, with the student tasked with making 
hypotheses, gathering data in the form of required experiments, and finally identifying seven different 
organelles contained in the cell. These tasks represent seven parallel goals (identifying nucleus, endoplasmic 
reticulum, Golgi apparatus, mitochondria, chloroplast, ribosome and vacuole) which together form the 
structure of the goal system in the ID module. Eigure 1 illustrates part of the goal system. Each goal is 
represented by a series of tasks to be completed. Through performing the tasks, students learn the 
scientific/deductive process to follow in order to confirm the identification of an unknown substance, in this 
case an organelle. Eor example, a student might be asked to identify the Golgi apparatus in the Virtual Cell. 
To complete this goal, three tasks must be performed, and the goal of identifying this organelle cannot be 
completed if any of the tasks are skipped or performed in the incorrect order. 
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• Task 1: hypothesize that an unkown organelle is a Golgi apparatus and scan it. 

• Task 2: perform the marker assay for glycosyl transferase to confirm the hypothesis. 

• Task 3: make a report identifying it as a Golgi apparatus. 
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Figure 1. Goal structure for the ID module: A goal is completed if its corresponding tasks are all completed. For example, 
to complete the “NucleusIDGoal”, two tasks need to be completed: the “DNAAssay” task which requires the player to 
perform the marker assay for DNA synthesis, and the “NucleusReport” task which requires the player to file a report to 
confirm the hypothesis. Note that the “DNAAssay” task must be completed before the “NucleusReporf’ task. 


2.2 ETC and Photosynthesis Goal Systems 

The ETC module introduces the player to the electron transport chain and cell respiration by first presenting a 
healthy mitochondrion, and then guiding the player to a damaged mitochondrion. The player is then required 
to repair the damaged ETC by accomplishing 3 tasks. 

• Task 1: investigate possible reasons for the cell's low ATP production; 

• Task 2: repair the damaged electron transport chain. 

■ Task 2.1: hypothesize the broken ETC complex. 

■ Task 2.2: leave the mitochondrion and find a ribosome. 

■ Task 2.3: purchase a new version of the broken complex. 

■ Task 2.4: re-enter the broken mitochondrion. 

■ Task 2.5: replace the broken complex with a new one. 

• Task 3: file an incident report 

There are six complexes involved in a functional ETC, as well as a healthy supply of the "raw material" 
substrates: succinate dehydrogenase, hydrogen, and oxygen. Any one of the complexes could be broken 
which would bring the ETC to a halt. As remediation, a substrate pointed at the proper complex will jump- 
start the ETC from that point up until the broken complex is encountered. 

The photosynthesis module is designed to teach one of the most important biochemical processes in 
plants. Its goal structure is similar to the ETC module except for the final task; the player is required to repair 
an inactive section of the chloroplast to produce ATP. The detailed goal structure of the photosynthesis 
module is shown in Eigure 2. 
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To make a game suitable for students with different backgrounds e.g. high school students and college 
undergraduates, the tasks or goals in the ETC and photosynthesis can be easily adjusted. For example, in the 
photosynthesis module, one task is to let students gather the needed substrates to improve the health of the 
cell. It is obvious that the more missing substrates, the harder the task. Many students know that CO 2 is 
necessary in the photosynthesis process. But that photons serve as tools to shake electrons free from 
chlorophyll, and move energy through the rest of the electron transport chain, is more esoteric knowledge. 
Therefore, the conceptual difficulty of the task of collecting necessary substrates could be decreased by 
adding protons to the student’s inventory at the beginning. 




Figure 2. Goal structure for the Photosynthesis module:A goal is completed if its corresponding tasks are all completed. 
For example, to complete the “ViewHealthyOrganelle” goal, the player must enter a healthy organelle first, and then view 
an educational animation. After that they learn how to use substrates to test organelles by firing an available substrate into 

the chloroplast, and finally exit the healthy organelle. 


3. INTELLIGENT TUTORS 

From the perspective of intelligent tutoring systems, the agents of interest must fundamentally support 
models of the knowledge of a domain expert and an instructor (Slator, 1999). We maintain software tutoring 
should not only consider current activities, but also should be aware of past performance, in order to give 
contextualized, individualized tutoring. 

3.1 Knowledge Base 

We maintain various information resources in the library of the game. First, based on a concern for 
information canonicity and coverage, we integrate the relevant biological knowledge derived from our 
content experts into the backend knowledge base. Students can easily use the toolbox provided in the game to 
navigate to the game’s online encyclopedia, using it to find desired information. Second, multimedia 
educational materials are incorporated to complement the system’s existing knowledge, e.g. the animations 
introducing the process of electron transport and photosynthesis. Students can actively study these materials 
at any point in the game. Or in another scenario where the player is stuck, problem-oriented and 
individualized knowledge will be offered as often as needed. This kind of knowledge, like non-obvious 
student play patterns, goes beyond domain-specific knowledge, and constitutes another significant aspect that 
helps in customizing the tutoring for individuals. In addition, mini-games that go along with each lesson 
module are developed for situations where students finish early or where they have trouble understanding 
lesson concepts. For example, the mini-game designed for the ETC module teaches ATP production by 
requiring the student to re-orient complexes to correct positions so that ATP can be steadily produced. 
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3.2 ID Module Tutors 

There are currently three modules in the Virtual Cell. The first is the Organelle Identification (ID) module, 
where students need to identify seven organelles correctly to accomplish their learning goals. During this 
activity, play history is recorded by the system which serves as important data for determining the type of 
tutoring to be provided at later stages in the game. 

3.2.1 Modeling Method for Capturing Learning Activities 

Table 1. Semantic Type - Concept Mapping 


Semantic Type 

Instances (Examples) 

Organelle 

nucleus, endoplasmic_reticulum, Golgi_apparatus, 
mitochondrion, chloroplast, ribosome, vacuole 

Correct Report 

correct_report_no_assay, 

correct_report_correct_assay, 

correct_report_incorrect_assay 

Incorrect Report 

nucleus_as_ mitochondria, 
endoplasmic_reticulum_as_ chloroplast 
vacuole_as_Golgi_apparatus 

Assay 

DNA_synthesis, succinate_dehydrogenase, 
phospholipid_biosynthesis, glycosyl_transferase, 
protein_biosynthesis, chlorophyll 


To model student play history, we define a student profile which is essentially a set of related concepts 
that together represent the student learning activities. This model is inspired by the closed text mining 
algorithm (Srinivasan, 2004). To further differentiate between different concepts, semantic type (ontological 
information) is employed in profile generation. Table 1 illustrates part of the semantic type - concept 
mappings. Here, each student profile is defined as a vector composed of a number of semantic types. 

profile{Stud,) = {ST„ST^,...,STJ (1) 

Where represents a semantic type to which the related concepts representing the student’s learning 
activities belong. Each semantic type can be further expanded by an additional level of vector composed of 
concepts that belong to this semantic type and relevant to the student’s play activity. 

STi ( 2 ) 

Where m. e 57] represents a concept under the semantic type ST. and vr, ^ denotes its weight. When 

generating the profile we replace each semantic type in (1) with (2). To compute the weight for each concept 
in (2), we employ a variation of the TF*/DT’weighting scheme (Jin and Srihari, 2006) and then normalize the 
weight under each semantic type; 


I highest{s. , ) (3) 

Where Z = l, 2,..., r and there are totally r concepts for 57] , ^ = the number of occurrences of , 

where nij e 57] . By using the above three formulae we can build the corresponding profile for any given 

student. To summarize, the procedure for building a student profile is composed of the following four major 
steps: 

• Step 1: Concept Retrieval: retrieve all relevant concepts from the student play history. 

• Step 2: Semantic Type Employment; each concept is associated with and grouped under one semantic 
type (e.g.. Assay, Incorrect Report) in which it belongs. 

• Step 3: Weight Calculation; for each concept, a variation of the TF*IDF scheme is used to calculate the 
weight (i.e., s- j as shown in Eormula 3). 
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• Step 4: Weight Normalization: the weight of each concept is further normalized by the highest concept 
weight observed for its semantic type as given in Formula 3. Within each semantic type, all concepts are 
ranked according to the normalized weights. 


3.2.2 Tutoring Strategies 
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Figure 3. The tutor’s decision making process for incorrect reports: Once an incorrect report is filed, the system will 
examine the player profile and check out the weight of the misidentified concept. If its weight exceeds the threshold 
defined in the system, either blind or oriented tutoring will be activated. 


The generated profile represents the student’s play history and potentially includes valuable patterns to be 
mined and utilized. Each time the student conducts an activity, e.g., performing a DNA synthesis assay on a 
nucleus or mistaking a mitochondrion for a chloroplast, the profile is updated (i.e., the weight of each 
concept in the profile is recalculated) to reflect the up-to-the-minute learning status of the student. The 
tutoring system makes tutoring decisions based on the discovered pattern frequency (i.e. weight computed 
using formula 3). If the pattern frequency does not reach the threshold predefined in the system, blind 
tutoring will be offered, otherwise oriented tutoring will be launched. Here blind tutoring means the tutor 
provides general conceptual information like the structure and composition of a plant cell, and does not 
further investigate the student’s learning problem (e.g., “did the student perform an incorrect assay?”). On the 
contrary, oriented tutoring indicates the tutor starts to explore the student’s past activities, such as “the 
student is mistaking what for what?” and then attempts to offer the best problem-oriented advice. 

For example, a student may have confusion between a nucleus and mitochondrion. If this mistake has 
been captured frequently enough to reach the threshold set in the tutoring system, a student-oriented tutoring 
session (with a specific focus on explaining the nucleus and mitochondria) will be activated. Figure 3 
demonstrates the detailed remediation strategies for incorrect reports. 


3.3 ETC and Photosynthesis Tutors 

We adopt similar tutoring strategies in the ETC and Photosynthesis modules since they have similar goal 
structures. The ETC occurs in mitochondria as the third and last stage of cellular respiration. Tutoring 
strategies for the ETC are illustrated in Eigure 4. 


114 










lADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2013) 



Figure 4. Tasks and tutoring opportunities in the ETC module: Tutoring is offered if a failed task completion is detected. 

For example, “sub-task 2.2” requires the player to buy a complex. If the player fails to buy the correct one (e.g., an 
incorrect one is bought), a tutoring message will be sent. 

However, appropriate tutoring opportunities might be difficult to recognize in some complex cases. For 
example, in the ETC module, the final task is to enable a broken mitochondrion to start transporting electrons. 
In order to activate the electron transportation, broken carrier substrates that hold electrons like NADH 
dehydrogenase must be replaced. Before that replacement, a sequence of activities need to be conducted: 1) 
check the current inventory to find a healthy complex; 2) buy a healthy complex if the current one is running 
out; 3) replace the broken complex with the healthy one. In this case, it is not certain all of the activities must 
be involved, which means one or more of them might not be needed based on practical considerations such as 
the current inventory status. 

Two alternative tutoring strategies are proposed to handle this issue. One is to wait until an incorrect 
substrate is chosen to replace the broken one, no matter whether the student inventory has the correct 
substrate or not. In this case, the student might get stuck if their inventory is running out of available 
substrates. The other is to check the student inventory before the replacement task is initiated. 


4. CONCLUSIONS 

This paper proposes a new solution to intelligent software agent tutoring that integrates data mining 
techniques with intelligent agents. This instructional system aims to individualize learning experiences 
through the incorporation of a data mining model based on student learning history. Blind tutoring is 
provided to meet the requirements of a majority of the students, while oriented tutoring is customized for 
struggling students. The integration of data mining techniques would also benefit other related tasks such as 
educational psychology and student-centered learning. 
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To support the development of such tutors, a comprehensive goal system that covers various pedagogical 
scenarios, targeted to focus the learning process on academic goals is presented. In this virtual environment, 
students demonstrate mastery of required knowledge and skills through the completion of these learning 
goals. 

The solution introduced in this work is implemented in an educational 3D game for biology students. 
Virtual Cell (Borchert et ah, 2013). Besides cellular biology, there is potential for adapting this solution to 
other applications that involve scientific reasoning and scientific methods. For example, the proposed data 
mining model can be re-used in other educational game arenas: psychology, math, physics and social science 
and humanities. The goal system supports cellular biology education and can also be extended to meet the 
needs of sub-disciplines like bacteriology, and many other specialized cells in multicellular organisms. 

For future work, the ontologies developed in this task can be further improved to fit more learning 
activities. Furthermore, more tutoring opportunities might be discovered by collecting and analyzing the logs 
generated from mini-games. And common patterns can be generalized to construct typical learning cases for 
library enrichment. We will be exploring these issues and evaluating their performance in our future work. 
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ABSTRACT 

In the face of budgetary constraints, new marketplace competition, digital innovation and an emerging generation of 
students with new demands and new needs, higher education in America is challenged as never before. This paper 
reviews the various challenges facing traditional non-profit educational institutions, considers the potential of an online, 
technology-mediated curriculum, reviews Constructivist Pedagogical Theory as a viable tool for faculty at traditional 
non-profit institutions and presents findings from a case study of a technology-mediated, hybrid course inspired by 
Constructivist Theory. 
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1. INTRODUCTION 

Online bullying, global warming, the Lehman Brothers bankruptcy, partisan political grid-lock, the collapse 
of the 2004 New York Yankees. . . .Perfect Storm Analogies have become popular tropes in assessments of the 
radical, overwhelming and sometimes life-threatening impact of an unforeseen confluence of forces on 
individuals, systems, institutions, and social groups. 

In the face of budgetary constraints, new marketplace competition and an emerging generation of students 
with new skills and new expectations, the Perfect Storm Analogy applies to higher education in America as 
never before. As Schuster and Finkelstein (2007) note; 

In the near-millennium history of the academic profession, there has never been a time when change is 
occurring so rapidly.... economic and technological mega forces... continue to remold academic 
practice, spanning pedagogy, scholarship, and all aspects of administering and managing colleges and 
universities (p. 57-58). 

Referring to the “tsunami” that threatens higher education, Stanford University president John Hennessy 
advices, “I can’t tell you exactly how it’s going to break, but my goal is to try to surf it, not to just stand 
there.” In that spirit this paper will 1) review the marketplace competition facing traditional non-profit 
educational institutions, 2) assess the emergence of a new generation of learners, 3) consider Constructivist 
Pedagogical Theory as a viable tool for faculty at traditional non-profit institutions to respond to marketplace 
competition and serve new learners and 4) present findings from a case study on the implementation of a 
technology-mediated, hybrid course. 


2. NEW ECONOMIC MANDATES 

Government support for higher education is at a twenty-five year low and last year support for state 
institutions fell 7.6% (Kelderman, 2012). Institutional responses to demands for fiscal accountability were 
predictable: university administrators have frozen tenure-track hiring, increased class sizes, cut funding for 
research and facilities, raised tuitions and hired more adjunct faculty. Some responses are troubling. Texas 
A&M University’s recent proposal to apply cost-benefit performance metrics on faculty is problematic to 
many in higher education. 
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Critics see this kind of review as a harbinger of more blunted, draconian measures that overlook other, 
significant features of faculty productivity (Simon & Bancharo, 2010). Gerbner (2001) warns that 
management and assessment models like these, if widely adopted, will “reduce higher education to a form of 
narrowly conceived job training” (p. 24) and will effectively undermine the essential values and standards of 
traditional academe. 


3. NEW PROVIDERS 

In the past decade new educational providers have been making “a meaningful difference” in the competition 
for students, and for state and federal support (Marklein, 2010). These publically traded, for-profit 
educational institutions (e.g. The University of Phoenix, Westwood College, Sanford-Brown College) are the 
fastest growing segment in higher education and currently serve nearly two million students. Cost- 
comparisons between the For-profits and Non-profits are difficult to accurately assess. (Shapiro & Pham, 
2010). It is clear, however, that For-profits can substantially control costs by 1) limiting investments in 
scholarship, service and infrastructure, 2) focusing on the profitable curricula to the exclusion of less 
profitable curricula and 3) maintaining a workforce heavily dependent on part-time faculty. 

Advocates for the For-profits contend that the asynchronous, online programs that dominate the curricula 
of these institutions are especially valuable in meeting the needs of non-traditional students; and that this 
alternative delivery system will eventually prove appealing to traditional students with promises of low-cost 
education, ease-of-use curricula and accelerated matriculation. The For-profits are in fact gaining significant 
purchase across the educational marketplace; student loans and federal grants are swelling the revenues of 
For-profits; California students in For-profits are receiving one-third of all Cal Grant support (Kelderman, 

2012) ; some For-profits are beginning to pursue regional accreditation; and some are buying up Non-profits 
that are already accredited (Calvert, 2010). Analysts predict that disruptive innovation (Christensen) in the 
form of “bottom-feeding” by For-profits will most directly threaten the long-term health of public 
universities. (Parry, 2011) 

The For-profits may be in a transition period that parallels IBM’s decade-long transition from hardware 
sales to info-technology and consulting services (Blumenstyk, 2011). From this perspective, For-profits will 
continue to market degree programs and will branch out to develop other forms of “educational product.” In 
fact. The University of Phoenix is offering new services that can be sold to Non-profits; and Cappella 
University is producing online curricula for corporate clients - software products that could easily be spun 
off to Non-profits. There are similar developments outside the educational arena. Corporate media interests — 
like Walt Disney Company, The Discovery Channel and News Corporation — have begun targeting the 
educational market as an arena for expansion. News Corporation chief executive Rupert Murdoch is 
“thrilled” with the prospect of generating revenue by providing digital teaching and assessment tools to all 
educational providers (Barnes & Chozick, 2012). 

Traditional colleges and universities also face competition from within their own ranks. MIT and Harvard 
have invested $60 million in support of edX, a non-profit provider of Massive Open Online Courses; 
Coursera, birthed by two Stanford professors, sponsors MOOCs at thirty-three different traditional colleges 
and universities, offering over two hundred courses (Friedman, 2013); and Western Governors University, a 
public, accredited online university offers BAs and MAs, targeting non-traditional students with “affordable 
competency-based” learning modules (Kelderman, 2011). 

The success of the For-profits has brought increasing scrutiny regarding tuition increases, graduation 
rates, loan defaults and recruitment practices. In 201 1 twenty-three states filed more than fifty bills calling for 
more comprehensive oversight of the For-profits and the for-profit model. F. King Alexander, president of 
California State University Long Beach, directly raises the issue of For-profits in his state “milking the 
system” (Kelderman, 2012, p. A4). Regarding MOOCs, completion rates are disappointing — often less than 
10% of enrollment — and traditional universities are hesitant to accredit MOOC -based courses (Kolowich, 

2013) . 
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4. NEW LEARNERS 

The current generation of high school seniors entering university life present yet another challenge to 
traditional educational institutions (Proserpio & Gioia, 2007). Many first-year students have unique skills, 
make unique demands of their education providers and will eventually enter a unique post-graduation 
marketplace. Simply put; this is the first generation of incoming students with personal technology as part of 
their social DNA (Verhaagen, 2010). This iGeneration is dominated by Digital Natives who take for granted 
a world characterized by constant, customized, portable, personal connectivity; and demand the same in their 
educational environs (Rosen, 2007 and Montgomery, 2007). 

Some recent, small sample research suggests that these earliest-of-adopters, spawned in a multi-tasking 
Petri dish of smart phones and Twitter accounts, are beginning to exhibit a subtle form of neural re-wiring: 

The growth curve on the use of technology with children is exponential, and we run the risk of being 
out of step with this generation as far as how they learn and how they think.... We have to give them 
different options.... (Rosen, 2007, in USA Today, p. 2A). 

They’re less interested in learning facts and learning data than in knowing how to gain access to it and 
synthesize it and integrate it into their life. . ..Their brains are developing in ways where they’re taking 
in astronomical amounts of information, screening out unimportant details and focusing on the parts 
they need (Verhaagen, 2010, in USA Today, p. 2A). 

New Economic Mandates, New Providers, New Learners - threatening swells in The Perfect Storm 
roiling on the horizon - are forcing faculty and administrators at traditional universities to “negotiate a new 
role in a new era” in hopes of preserving the core values and ensuring the special nature of the academy in 
American culture. Admittedly, there is “a complicated mix of benefits and costs” to be reckoned with in 
these negotiations (Schuster & Finkelstein, 2006). Historically, colleges and universities have been charged 
with conserving cultural memory, creating knowledge, and mentoring generations of learners in 
analytical/practical skills. To this end, educational institutions have focused on three primary domains: 
students, curricula and scholarship (Christensen & Eyring, 201 1). It’s clear that — in the face of The Storm — 
the charge, the focus and the domains need to be re-considered, re-calibrated and re-defined. Fashioning new 
curricula to address a generation of Digital Natives is essential if traditional colleges and universities want to 
remain competitive in the shifting educational marketplace. 


5. CONSTRUCTIVIST THEORY 

In terms of serving students and evolving curriculum. Constructivist Education Theory offers intriguing 
strategies for educators in the non-profit realm. Inspired by Plato’s notion that “genuine education” entices 
students to search rather than memorize, the constructivist credo celebrates a new “transformative 
relationship” between students and teachers that requires innovative pedagogical strategies that drastically 
shift the emphasis from top-down practices to shared governance in the learning process. (Buckman, 2007). 
Hawk (2005) argues that constructivist tactics are uniquely suited for engaging what he dubs “post-modern 
learners.” To this end, faculty need to reject dated notions of sage-on-the-stage academic punditry and adopt 
the role of maestro-like facilitators who effectively orchestrate and maintain flexible “communities of 
practice” that foster indigenous, active, collaborative learning environments. Hirtle and Smith (2010) 
demonstrate how enacting a technology-mediated curriculum that exploits online access, web-based software 
and personal technologies can engender “deep thinking.” 

Engaging the Digital Natives by exploiting the technology they already embrace makes sense. Building 
on McLuhan’s notion of mediums-as-messages. Brooks & Brooks (1993) encourage teachers to be creative 
regarding media used in the curriculum. They insist that before teachers encourage autonomy and initiative 
by students, they should ascertain students’ abiding skills and adjust pedagogical strategies accordingly. 
Similarly, Short & Reeves (2009) insist that teaching is most effective when content is made available in 
mediums “fitted to and understood by” all participants. Early childhood educators have recognized this 
challenge for decades. An early proponent of Constructivist Theory, Constance Kami (1983), uses a 
horticultural analogy to illuminate successful teaching; “Plants grow not by addition of new parts from the 
outside but by differentiation and coordination of new parts from the inside.” 
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Educators in business schools have been especially creative in this regard, employing poetry (Morris, 
Urbanski & Fuller, 2005), film (Bumpus, 2005), TV game shows (Sarason & Banbury, 2004), graphic novels 
(Short & Reeves, 2009) and Web Sites (Gossett & Kilker, 2006) to engage their classrooms. In regards to 
student engagement, the distinction between education, communication and entertainment may be moot (see 
McLuhan, 1960). In this spirit, bell hooks (1994) observes that “the classroom should be exciting, but there 
is little discussion of that matter in higher education - it is viewed as potentially disruptive of the atmosphere 
of seriousness assumed to be essential to the learning process.” 

This theoretical perspective recognizes that talented faculty bring something special to the educational 
process (critical analysis, knowledge creation) and that the promise of technology-based education offers 
something equally special (familiarity, collaboration). One way to exploit the respective strengths of 
traditional faculty and emerging techno-mediated curricula is to design so-called “hybrid” or “blended” 
classes. 


6. HYBRID CLASSES 

Hybrid classrooms combine the benefits of traditional face-to-face instruction with the benefits of online 
learning. Hybrid classes conducted by talented faculty trained in creating such learning communities — who 
also engage in substantial research activity — can offer students a service that the For-profits cannot match. 
In designing hybrid learning enviornments faculty must re-consider the nature of the Digital Natives they 
recruit, engage and ultimately serve. Educators have long promoted the notion of nurturing “active learners” 
but the term has been vaguely defined and may have limited use in regards to today’ s students (Proserpio & 
Gioia, 2007). Dismissive notions of multi-tasking, attention-challenged, tweet-mongering teenaged 
barbarians hacking at the gates of the academy are not helpful - nor entirely accurate. Wolf (1996) for 
instance suggests that faculty should view the evolving media habits of today’s students - often dismissed as 
a generational failure - as a natural, instinctual response to an environment where constant info-stimulation 
has become the norm. 

Ideally hybrid classes will “speak to” the Digital Natives in a language they hold dear while 
simultaneously leavening the educational discourse with unique features that only a proven teacher- 
researcher can inject into a learning community. Kenneth E. Hartman, president of Drexel University Online 
insists that the future of education will be characterized by “new mixes” of online and in-person teaching 
(Young, 2011). Christensen and Eyring (2011) predict that expertly fashioned curricula with online, techno- 
mediated hybrid courses will bring a “learning renaissance” to higher education. 

Gameramn’s (2006) description of high school classes that encourage the free use of technology in class 
was instructive. During in-class tests students were allowed to access the Internet, to text and to use 
analytical software. In the spirit of open-book or take-home exams, free access to information sources was 
exploited as a means to test to higher levels of competence. And, in the spirit of calculator-use in 
mathematics classes (circa 1960), open use of technology during tests allowed students to more efficiently 
process information and streamlined the completion of redundant/mundane tasks. 

Informed by classroom experiments like these, in the spring of 2010 I adopted similar pedagogical 
strategies in a hybrid class in Mass Communications at a mid-sized state university. This paper will 
document a modest attempt to 1) enact the hybrid curriculum, 2) encourage students to participate a unique 
techno-mediated learning community and 3) re-calibrate my own role as teacher. 


7. THE HYBRID EREE RANGE CLASSROOM 

MCOM 491 The History of New Technologies is an advanced seminar class designed for undergraduate 
seniors majoring in Mass Communication. The class examines theories of innovation and the evolution of 
communication technologies. In the previous versions of MCOM 491, students were assigned a text, read 
articles posted on the university’s Blackboard site, engaged in Discussion Forums on this site, submitted 
Email Assignments, took three tests and presented a group project. 

In Spring Semester 2010, I adjusted the pedagogical strategy to create a hybrid version of the class. 
Students were encouraged to become Free Range Students by using personal technology at any and all times 
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- during lectures, screenings, presentations and tests — as long as they did not make noise (cell phone 
conversations, computer games with sound on, etc.). In lieu of a class project, students would construct a 
small-group Wiki Project; all other tests and assignments were, save for minor updating of materials, 
identical. The class was usually limited to 20 students. MCM students heard about the proposed syllabus by 
word-of-mouth. Twenty-five students registered for the class. The class met from 9:30 to 11:50 on Tuesdays 
and Thursdays. Eight times during the semester, the class did not meet on site but “met” online in real-time. 

All undergraduate classes in this department require student evaluations - IDEA Diagnostic Eorm Reports 

- that are public record. Students were reminded that their responses on the IDEA forms were public record 
and were encouraged to respond to both closed and open-ended evaluation questions on these forms. In 
addition, students were informed that I would closely observe their use of technology in the classroom. All 
students indicated they were comfortable with the hybrid arrangements. 

—Text: Communication in History by Crowley & Heyer, 6* Edition 
—Handouts - various 

—Lectures - Growth of the Internet, communication technologies, theories of innovation 

—Electronic Reserves - chapters from previous editions of the text 

—In-class screenings - Internet history, technological innovation 

—Guests Speakers - Wiki experts from the university’s Computer Services Department 

—Live Email Assignments - self-reports on personal use of technology 

—Regular postings on Blackboard Eorums: related to readings 

—Three tests: one multiple-choice, one short answer, one long essay 

—In-depth critical analysis of a Web Page (cleared with instructor) posted on Blackboard 

—Group Wiki Page (selected from a list of topics specified by instructor) 

All tests were administered in the classroom. During testing, students could use: the text, the handouts, 
the Blackboard site, the Internet and personal technology. No talking on phones. No talking to classmates. 


8. DATA COLLECTION 

A case study approach was employed 1) to assess this first attempt at a hybrid classroom with students 
“ranging freely” with personal, mobile devices and 2) to compare this class to previous versions of the class. 
The intrinsic case study method (Stake, 1995) was deemed appropriate for assessing, from multiple 
perspectives and with various measures, this initial implementation of a techno-mediated learning community 
that required a re-calibration of student-teacher relations. I compared student evaluations and grades from the 
hybrid class and from a previous version of the class. I systematically recorded observations of in-class 
participation and in-class behavior and compared to the previous version of the class. I compared in-class and 
online participation, test grades and term projects in both classes. I reviewed my own role as teacher in the 
hybrid environs and compared this experience to my previous experience with the class. 


9. OBSERVATIONS AND RESULTS 

The Tests: 

Average grade on multiple choice test = 88/100 
Average grade on short essay test = 78/100 
Average grade on long essay test = 75/100 

Student Activity During Tests (Instructor observation and unsolicited student comments): 

2 students set up chat rooms for communal use. 

16 used laptops, 2 used cell phones, 6 used laptops AND cell phones, 2 used Skype. 

1 student did not use any technology in the class. 

15 used search engines during tests. 

14 engaged in online debates about answers to multiple-choice questions. 

23 referred to the book and hand-outs during the test. 

7 students teamed-up before the test to coordinate technology. 


121 



ISBN: 978-989-8533-18-0 © 2013 lADIS 


Student responses to testing (as recorded on IDEA Forms): 

A new kind of classroom. 

Great having to work together. 

I liked asking others why they thought an answer was correct. 

Gives class a way to think together as a team. 

Stimulating. Made me engage in debate. 

Not too easy because the info from others makes you question your decisions. 

Nice how we could reason things out. But it took longer. 

I dig it but it took longer. 

Not everything in the chat room was correct. 

So many people discussing, it posed a doubt as to who is wrong or right. 

I liked the idea of figuring out a problem with someone else. 

I liked it but it stressed me out. 

Use of personal tech during a test is a crutch, allowing students to study less. 

Thanks for letting us use own technology. 

I didn’t care for this style. Class didn’t always follow syllabus. 

Assessment of class environment (Instructor’s observations): 

To varying degrees all Students checked Facebook, Emails, surfed & chat-roomed 
3 students studied for other classes on their laptops 

All students to varying degrees used technology to record notes from the class 
3 students used laptops in class to re-write resumes 
3 students used laptops in class to edit videos 

19 students used personal technology in class to check follow-up on class content 

Student responses to classroom environment (as recorded on IDEA Forms): 

We’re gonna do this anyway, might as well make it legal. 

It’ s nice to be treated like an adult. 

I really like being able to multi-task. I’m getting better & better. 

My internet didn’t work all the time. 

The way other students wack away on their technology bothers me. 

I love it! And feel as tho I still pay attention. 

It allows us to stay in touch w/other people and ask questions. 

Our generation knows how to multi-task. 

I think the point is for students to learn, and w/new tech, this is the way we do learn. 

It was closer to what you would do to learn something in a non-classroom setting. 

If we were confused about something you said we could ask each other. 

It’s very disrespectful, an odd situation. 

Gives the class a way to think together as a team. 

When I get a real job. I’ll do this all day! 

Instructor’s Response: 

I was humbled: students often ignored lectures, requests for comments and questions-to-the-class. 
During screenings students opted to use personal technology in lieu of viewing. 

Responses to questions put to the class were delayed, but more accurate and comprehensive. 
Compared to previous versions of this class, students took longer to complete tests. 

The Wiki Projects were, overall, disappointing. 

This hybrid class required significantly more time-investment on my part. I needed an assistant! 

I was often surprised and impressed when the class “took hold” of a discussion. 
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10. CONCLUSION 

This attempt at a hybrid, computer-mediated-curriculum was a qualified success. Most students approved of 
“ranging freely” with personal technology during class and during tests. Final grades were similar to final 
grades in previous versions of the class. Grades on the (one) multiple-choice test were significantly higher in 
the hybrid class. 

Quantitative student evaluations were slightly higher than those from previous versions of the class. 
Open-ended evaluative responses were more detailed in the hybrid class. The five Group Wiki Projects were 
disappointing: all lacked structure, all included a number of “dead” links, only one effectively employed 
streaming video and only one included original animation. Compared to in-class participation in previous 
versions of the class, there was more in-class participation in the hybrid class. Compared to previous versions 
of the class, in-class participation in the hybrid class — though often delayed and stuttered - was significantly 
more comprehensive and challenging. I attribute this to student use of personal technology to supplement 
class material, to challenge my assertions and to engage in collaborative fact-finding. Online student postings 
in the hybrid class were not significantly different than online student posting in previous versions of the 
class. 

As a teacher, I was forced to accept unfamiliar interactions with students and unfamiliar actions by 
students. Occasionally, I felt a loss of control. At times, the class took control - by using personal 
technologies — to consider and debate class-related issues. At times, the class took control - by using 
personal technologies - to engage in extended conversations about issues unrelated to class content. In both 
scenarios I was a bystander. 

Student evaluations, classroom observations, online observations, answers on tests and submitted 
assignments indicated that at various times, for varied lengths of time and sometimes in unpredictable ways, 
a unique, virtual learning community came in and out of existence in the hybrid class. In most class room 
meeting, students engaged in active, indigenous collaborative learning. 

Informed by this experience, I plan to (once again) re-calibrate pedagogical strategies in MCOM 491 The 
History of New Technologies. In the next version of the class, the ranging will not be as free and the group 
projects will be more closely monitored. Specifically, free-range use of personal technologies will be allowed 
only during selected class times (testing will remain free-range). I will personally take hands-on direction 
over the building of Wiki Projects - including in-class instruction and group-specific instruction. The next 
hybrid class will be assessed in terms of student evaluations, test results and personal observations and I will 
re-consider the notion of Free Range use of personal technology in MCOM 491. 

Finally, considering the results of this modest proposal, it appears that constructionist pedagogical 
strategies combined with a computer-mediated-curriculum in a hybrid class have significant potential to 
respond to the needs of Digital Natives. Important questions remain: Can classes like this one be successfully 
developed, supported and promoted by traditional universities as a means of competing with New Providers 
in the educational marketplace? Can curriculum innovation like this be scaled to help traditional universities 
respond to the demands imposed by new economic mandates? Will faculty accept a new role that 
significantly re-defines what they do as teachers? Will university administrators reward and support faculty 
who take on this challenge? There is growing evidence that the students are ready - ready for a change they 
can really believe in. 
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ABSTRACT 

Based on experiences in teaching service design in a blended learning context, we developed an electronic learning 
environment (ELE) including features that turned out to be suitable for learners from different cultures. We used this ELE 
in Italy and in China. Students were guided through collaborative learning and mutual teaching. Students were supported 
by the teacher with the help of didactic technologies in a learning process that merged theoretical understanding with 
design for real life applications in their culture. In both student groups, our intended improvements seem to be 
acknowledged by the students’ opinions. 
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1. INTRODUCTION 

Our research on ELE is performed in practice, during teaching and working with students. The only type of 
research possible in this circumstance is action research: we analyse problems, we consider relevant literature 
as well as the results of our own previous design, we plan improvements in our concepts of ELE and we 
apply them. Consequently, we assess our actions and start a new cycle. 

The current report is an account of another cycle regarding the support of blended learning in the domain 
of service design. Previously (Consiglio and van der Veer, 2011) we discussed the intersection between 
technological innovations and adoption in society for the purpose of adult learning. Our aim was to develop 
an e-learning environment, to be available both as a standalone-learning marketplace and as support for 
classroom-based learning. Our intention was to use the open source process to improve the quality of 
learning anytime and anywhere and make it as flexible as possible towards the culture, learning style and age 
of the learners. Van der Veer, Consiglio, and Benvenuti ( 2011) showed how to adopt features in the ELE 
starting from the students’ goals, in order to support them (and the other stakeholders in the learning process) 
to work in a real life context. These studies were a first attempt where the development of an ELE took place 
in a single context, a class of about 26 bachelor students in a Curriculum of Architecture and Design of an 
Italian University in the city of Alghero (Sardinia). 

We took the chance to continue our action research approach by teaching the same course in a new 
context with an unknown cultural component: a group of 10 master students in a university in China in the 
city of Dalian, starting from the same ELE concept and the same learning resources. Our intention was to 
explore whether one can design a flexible ELE suitable for teaching in different educational cultures. We 
discuss our experiences with both cases where the same teacher adopts this technological resource to teach 
Service Design to Italian and Chinese students. 


2. ICT AND E-LEARNING, OPPORTUNITIES EOR HIGHER 
EDUCATION 

Our focus is on University level education and adult learning. At this level, learners and students are able and 
willing to set their own learning goals ( Jones et ak, 1994 ). 


125 



ISBN: 978-989-8533-18-0 © 2013 lADIS 


As far as they participate in a curriculum, the teacher and the educational institute will set learning goals 
as well. In well-designed education, all these learning goals will be consistent and motivating ( Williams and 
Williams, 2011). The type of e-learning that we consider is a learning process where the teacher, the school, 
and the student all agree on the main goals, and where the students have specific goals related to their 
personal interests and context. 

The development of information society gave rise to dynamic changes in the different tools and 
technologies available for support of the learning process ( Redecker et ah, 2009) . A couple of decades ago 
the idea of education provided through the Internet was only just in the beginning. Nowadays e-learning is a 
widespread practice. E-leaming techniques allow delivering educational content through the Internet. For the 
user-learner this represents a flexible learning solution, highly customizable and easily accessible. For the 
user-content expert or the user-teacher this represents an equally flexible solution for collecting, formatting, 
structuring, updating, and maintaining learning resources. In this vision e-learning environments cover a wide 
range of resources, practice and training applications, and virtual classrooms (Sampson et ah, 2002) 

In rare cases it seems useful to replace traditional education by e-learning, e.g. when geographical 
distance or time-zone differences prohibit face-to-face teaching. In many cases it seems useful to 
complement traditional classroom teaching with e-learning, since it allows learners to (partly) work in their 
own pace, time, and context, to choose for individual or group learning, to decide on the amount of practice 
they need or the sequence of studying content. This combination is often labeled blended learning (Garrison 
and Kanuka, 2004). 

Developing e-learning requires the combination and interaction between learning activities and teaching 
activities through electronic media. An e-learning environment should provide up to date resources as well as 
technologies, have a high level of usability and, above all, be adapted to different modes of learning, like 
inquiry-based learning or collaborative learning. 

Effective teaching and learning through e-learning depends on many factors. We will discuss the 
educational and management viewpoint and refer to the barriers and preferences related to time and location. 

2.1 Educational Viewpoint 

The development and implementation of the e-learning part of a blended course should consider at least three 
closely related fundamental aspects: course structure; didactic methodology; and planning of different 
learning activities 

The teaching strategy should consider that each student is different both in terms of cognitive and 
experiential learning and in availability of personal learning time (Boettcher, 2007 ). ICT-based interactivity 
may in fact accommodate the variable individual needs of teacher and student and the needs of 
communication and collaboration between teacher and student, as well as among students. It also may 
support flexible management of educational activities and flexibility in choosing place and time. 

ICT allows a flexible combination of, and alternation between, synchronous and asynchronous 
communication, allowing all stakeholders to communicate with each other in real time through the use of 
tools such as chat or videoconference, as well as to participate individually at will any time using forums, 
blogs, wikis, or e-mail. Providing this multitude of opportunities for communication enhances participation, 
collaboration and involvement in the learning environment. 

2.2 Management Viewpoint 

E-learning allows extensive use of multimedia for the content to be delivered (audio, video, web pages, 
podcasts, etc.) and of environments suitable for learning management (LMS, Learning Management System) 
or content (LCMS, Learning Content Management System). 

2.3 Barriers and Preferences 

To overcome barriers of time and place that prevent access to education, students can attend courses they 
need or like even in remote areas far from universities, or at time available for those with full time jobs or 
who live in remote time zones. 
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E-learning also allows students to choose their own preferred moments (just-in-time learning) and their 
own pace, a valuable commodity in the case of students who must balance learning with work commitments 
and / or family. 

2.4 Opportunities 

New technology provides opportunities for, and triggers, modification of the methodological approaches and 
of the roles of teachers and students within the educational process. We take a constructivist perspective 
according to which learning is a dynamic process, which takes place either through the active engagement of 
the student or through interaction with others (Bruner, 1960). With e-learning opportunities the teacher's role 
increasingly develops into being a facilitator and learning tutor, an expert in communication, and a manager 
and monitor of knowledge acquisition, while at the same time fostering socialization and group dynamics. In 
this way, the teacher (or learning resource designer) helps students to build their personal knowledge and to 
contribute actively to the shared knowledge of the group. E-leaming is effective when teacher-tutors, content 
experts, and students develop an interdependent creative and productive relationship. 


3. THE COURSE DOMAIN: SERVICE DESIGN 

Services are different from products that can be sold. Services are being provided and at the same time being 
used. After the service is provided the client does not own it, even if the service has been paid for. Production 
of the service and making use of it occur at the same time. 

Service design means planning and organizing the different providers, the infrastructure, and the relevant 
communication. Relevant and appreciated services are often based on organizing multiple stakeholders who 
all contribute to the total service. A well designed service will provide, both a needed and appreciated help 
for the clients of the service, and a positive experience of being helped in a way that fits the clients’ context 
and actual needs. 

The activity of designing service was originally considered as part of the domain of marketing and 
management. Shostack (1982) proposed the integrated design of material components (products) and 
immaterial components (services). 2004, Service Design Network was launched (www.service-design- 
network.org). Erom this we learn that service design can involve the design of artifacts (physical and non- 
physical) as well as the organisation of communication, of the situation and environment and of ways to 
provide and to use the service. Because the actual service exists (only) at the moment of provision and use, 
designers can not exactly specify them: service design only can suggest scripts to the stakeholders and users 
involved. Service Design requires: identification of the stakeholders, including users; definition of the 
requirements for the service and the organizational structure; description of service scenarios with roles for 
the stakeholders; and representation of the service to communicate to all stakeholders and to guide the 
provision and use during the actual service. 


4. THE CONTEXT: TWO CITIES 

The first instance of our course in service design was taught in the spring semester of the year 2009/2010 to 
30 bachelor students at the faculty of Architecture, University of Sassari (Italy), in the town of Alghero. In 
Alghero there is a tradition of guest students in the group, who manage by trying to communicate in Italian. 
English, spoken by guest professors, will be understood though speaking or writing in that language is 
somewhat problematic for part of the students. In our case the Dutch teacher was physically present 10 hours 
(in 2 days) every fortnight during the semester long (250 hours) course while in the remaining time students 
worked in teams of 4 or 5, and submitted their homework by email. The University provided an Italian 
speaking tutor who attended all classroom meetings and was available for the students at scheduled times 
when the teacher was not in the country, to support the students in understanding the learning resources 
provided by the teacher and the slides of the lectures. 
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Our course was structured along generally accepted approaches of user centred learning, adapted to the 
domain of Service Design, where collaboration with different types of stakeholders with varying goals, 
cultures, and professional visions is a main challenge. At all stages of the course, and all phases of the design 
process, we asked the student to consider and elaborate 3 aspects: (1) the context of current activities, 
including all relevant issues related to stakeholders; (2) the design space with all design question to be 
answered, all possible options, and all relevant criteria; and (3) creativity in considering ideas as well as 
combinations of ideas from all stakeholders concerned. 

The general design method introduced by the teacher was based on DUTCH (Design for Users and Tasks 
from Concepts to Handles; Van der Veer and van Welie, 2003). 

During this course face-to-face meetings between teacher and student occurred at relatively sparse periods 
when the teacher was in town. Alternative communication was by email. In addition, some of the students 
could not always be present when others presented work in progress, and expressed the wish to be able to still 
view their peers’ presentations. We identified several issues that required improvement, related to the fact that 
synchronous communication only was possible during a small part of the time: during the course period, the 
students repeatedly showed a need for a preview of the structure for the remainder of the course, as well as a 
need for reviewing parts that were discussed before. Also, the students told us it would be appreciated if all 
content, as well as pointers to additional resources, could be found at a single central location that would be 
accessible any time. Summarizing: the students hinted at a central website for both review and preview as 
well as for all additional resources and pointers. 

4.1 A Pilot Electronic Learning Environment 

In the next iteration of the same course (academic year 20II/20I2) 25 students were involved in a blended 
course. Based on the first empirical results we developed practical guidelines for the ELE, for the interaction 
of teachers and instructional designers with the ELE, as well as for the structure and format of learning 
resources in it. In order to support students to find their way in the ELE, a concept map was used to structure 
the digital environment, based on the lesson plan used in the previous year face-to-face course, see Eigure 1 . 
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Figure 1 . Concept Map of the ELE that Reflects the Structure of the Course 


The structure of meetings with the teacher, teamwork, and availability of a tutor was identical to the 
previous version. The videos of the classroom meetings were uploaded to the ELE immediately after they 
had been captured see Figure 2. The alternating classroom meetings, team meetings, and the (individual as 
well as team based) use of the learning environment resulted as an opportunity for blended learning. 

Another instance with roughly the same version of the course on service design was taught in China, at a 
group of 10 master students at the Dalian Maritime University, in October 2012. One of the students was not 
Chinese, there was a history of international guest professors, and all were used to speak English in the group 
and during lectures. In this case, the teacher was available during 7 consecutive days (including weekend 
days) for periods of 2 hours while the students were supposed to (and actually did) work for about 8 hours or 
more (60 hours in total). The ELE was improved based on or analysis of the previous version in Italy. There 
was no tutor available, and the students were completely happy to deal with the language issues involved. 
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Figure 2. Lecture Videos Were Available Immediately at the ELE 


5. THE COURSE WITH ELECRONIC LEARNING ENVIRONMENT 

5.1 Global Approach 

Like in the first version of the course (not supported by an electronic learning environment), the teacher left 
most of the teaching to the students. In fact he only explained a small number of service design techniques 
and tools giving pointers to resources, and each student got the task to find the best description of the other 
tools and techniques the teacher pointed to and to teach to the other students why and how to use these, the 
benefits, issues and problems, and the conditions for application. The students’ presentations were put on a 
dedicated YouTube channel as a resource during the rest of the course. In order to stimulate the students to 
improve their mutual teaching, some excellent student presentations were identified, and the students got the 
assignment to review these and to analyze why this examples of teaching made sense to them, both from the 
content point of view and from the presentation (i.e., knowledge sharing) point of view. 

5.2 Designing a Flexible Electronic Learning Environment 

We designed the supporting ELE in two parts; technology and content. The first part is the technology of a 
standard learning environment, the second part is formed by the structure of the content by the learning 
materials. The service is designed to inform, inspire and facilitate the students in their classroom based 
learning, collaborative learning and in free individual learning. The content structure and format complement 
the efforts to provide user centered interaction design, offering a holistic learning experience. 

Our system was structured based on the lesson plan used in the previous year for a fully classroom based 
learning process. It was expanded with additional opportunities for exploration, communication within teams 
and between multiple teams and teacher. We provided additional resources like online exercises and multiple 
different modalities of presentation of knowledge. E.g., we developed mini lectures (10 minute long teaching 
of a single technique like Cognitive Walkthrough, Moodboard, Persona) made available in different 
modalities; a) full text with pictorial illustrations; b) video recordings of actual teaching and c) slide shows 
with voice-over. 

The alternating classroom meetings, team meetings, and the (individual as well as team based) use of the 
learning environment supported integration with online learning activities, resulting in opportunities for a 
blended learning process. Activities to build the learning service were diverse, requiring a variety of skills of 
the people involved. A close collaboration between teacher and instructional designer is needed particularly 
during the development phase. 
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They need to match the instructional design of the classroom-based sessions with the online learning 
activities. 

During the course all lectures of the teacher were recorded on video and published embedded in the 
learning environment in addition to the presentation slides, to complement the notes that the students made 
during the lectures. This was especially important in this course because the lectures and class discussions 
were in English while the native language of the students was Italian or Chinese (in some cases another 
language like Spanish or Finnish). The recordings were additionally published on a YouTube dedicated 
channel to make them available on devices like smart phones. Special attention was paid on the way to 
structure the slides, the readability of text, and the visibility of face and gestures of the teacher in the video 
version. 

In the case of the Dalian course we made slight adjustments to publish movies because YouTube is not 
available. Therefore we uploaded the video lectures on a private server and deliver the movies embedded in 
standard web pages. 
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Figure 3. Forum in the ELE on Student Request, Allowing Discussion Prior To A Student Presentation 

5.3 Adaptation to Each Individual Class is Needed 

The way to prepare, to present, and to discuss sources for the learning environment requires special attention: 
classroom communication as well as on-line resources featuring in an actual individual course may have to 
be re-used later (life lectures turn out to survive as YouTube clips, short PDF files, voice-over presentations, 
citations in student generated learning resources, etc.) but other resources have to be personalized basing on 
needs of the classroom. Chinese students asked to be allowed to upload their presentation before the lecture, 
in a plenary forum where their peers and the teacher could discuss beforehand (see Figure 3) while for Italian 
students it was more suitable to have a dedicated space where they submitted their work individually as a 
design group to the teacher only, before the discussing it in the class. 
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Consistent with our constructivist perspective on learning, and based on request from some Italian 
students who could not attend classes where they were supposed to present, we decided to allow them to 
submit a home recorded video presentation of their mini lecture to the ELE. 

5.4 Student Opinions on the ELE 

At the end of both courses on Service Design that used the ELE we asked the students to answer a 
questionnaire. Eor the Italian students, we translated this questionnaire in Italian, the Chinese students were 
happy to answer English questions. The number of students that participated in the questionnaire 10 Chinese 
students and 13 Italian students) is too small to allow any statistical tests that would allow generalizations 
beyond the students in the two courses that we actually observed. However, the results certainly provide us 
with an interesting picture. Table 1 provides the answers that the students gave our list of questions. 


Table 1. Survey Answers for Two Course Groups 


Course group: 

China (#10) 

Italy(#13) 

a. ELE did help during course? 



- yes 

9 

6 

- a litde 

1 

6 

- no 

0 

1 

b. ELE useful for: 



- finding teacher’s slides? 

10 

8 

- viewing teacher’s presentation videos? 

3 

8 

- finding URs for extra information? 

8 

7 

c. wish further resources in the ELE? 

9 

7 

d. did you watch video presentations: 



- of other students? 

5 

12 

- your own? 

6 

10 

e. what did you learn from watching peer presentations: 



- to make readable slides 

not 

6 

- to speak to an audience 

applicable 

2 

- to structure the presentation 


6 

f. ELE feasible for smart phones in the future: 

6 

9 


Our Chinese group was more positive overall on the help that the ELE provided (question a). This may be 
related to their educational level (this where master students in their final year) as well as to their fluency in 
English. It may also be caused to improvements in the course website (we obviously continued to structure 
the ELE based on our experience with the previous course in Italy). Regarding the specific types of use 
(question b) we only identified one difference, the Chinese students in our group were less interested to 
review the teacher’s presentation video. Our group of Chinese students was more eager than the Italians to 
find additional resources (question c), possibly related to their educational level as suggested above in 
relation to question a. Our group of Italians systematically watched their peers’ presentations again (question 
d), probably because we asked them to do this in our attempt to have them reconsider their presentation 
performances. Eor the Chinese group we did not do this, since in fact we did not find too many good 
examples among the group to start with. In fact we provided some examples ourselves, telling them how we 
acted in presentation and why (“make sure you look at the audience, that helps make them pay attention, like 
I just am showing you now’’). Because we made a special effort on making the Italian students aware of the 
presentation skills of some (in fact the best) of their peers, we asked our Italian students what they did learn 
from the other students’ video presentations (question e). It seems only part of the students felt they learned 
something from their peers’ examples. However, both authors, as well as the Italian tutor at this course, were 
convinced the presentations improved significantly for the large majority of the students in this group. 

Since in both student groups smart phones seemed to be a natural extension to the students’ hands we 
asked them if they would like to use smart phones for courses in the future (question f). Both groups showed 
a majority who thought this might be expected. 

As stated at the start of this section, we would not dare to generalize. But surveys like this help us to find 
what worked in our current cases and helps us understand what the effect our effort is. 
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Based on that we make our plans for a next instantiation of our ELE. That is the essence of action 
research. And the difference between the Italian and the Chinese context did not seem a major source of 
different student behavior. 


6. CONCLUSION 

Our constructivist view on higher education and our action research approach towards iterative design and 
assessment of an ELE shows how new ICT may be applied to provide blended learning, adapted and 
adaptable to cultural as well as to individual context and learning needs. Service design was just an example, 
and in fact we are applying the same approach to other learning domains like task analysis, visual design, and 
the design for cultural heritage support. But that’s another story. 


REFERENCES 

Boettcher, J. V., 2007. Ten core principles for designing effective learning environments: Insights from brain research 
and pedagogical theory. Innovate, 3(3). Retrieved May 20, 2009, from www.innovateonline.info/index.php 

Bruner, J., 1960. The process of education - Harvard University Press. 

Consiglio, T. and van der Veer, G.C., 2011. Designing an interactive learning environment for a worldwide distance adult 
learning community. In; Anke Dittmar & Peter Forbrig (Eds) Designing Collaborative Activities - Proceedings of 
ECCE 2011. Rostock. ACM Digital Library, pp.225-228. 

Garrison, R. and Kanuka, H., 2004. Blended learning: Uncovering its transformative potential in higher education 

The Internet and Higher Education Vol. 7, No. 2, 2nd Quarter 2004, pp.95-105. 

Jones, B., Valdez, G, Nowakowski, J., & Rasmussen, C., 1994. Designing Learning and Technology for Educational 
Reform. Oak Brook, IL: North Central Regional Educational Laboratory. 

Moritz, S., 2005. Service design - Practical access to an evolving world. Koln International School of Design (KISD), 
Kbln, Germany. 

Novak, J.D., Gowin, B.D., 1984. Learning How To Learn, Cambridge University Press. 

Novak, J. D.,1999. Learning, Creating, and Using Knowledge: Concept Maps As Facilitative Tools in Schools and 
Corporations, Lawrence Erlbaum Associates. 

Novak, J.D., Canas, A., 2008. The theory underlaying Concept Maps and How to Construct and Use Them, Institute for 
Human and machine Cognition, Pensacola. 

Prime Minister’s Strategy Unit., 2007. Building on progress:Public services. Prime Minister’s Strategy Unit, London, 
UK. 

Redecker, C., Ala Mutka, K., Bacigalupo, M., Ferrari, A., Punie., 2009. Learning 2.0. The impact of Web 2.0. Innovations 
on Education and Training in Europe - Final Report - JRC Scientific and Technical Reports. 
ftp://ftp.jrc.es/pub/EURdoc/JRC55629.pdf ( accessed 2/28/2013 ). 

Rogers, E. M., 1995. Diffusion of innovations (4th ed.). New York: Free Press. 

Sampson, D., Karagiannidis, C., and Kinshuk, 2002. Personalised Learning: Educational, Technological and 
Standardisation Perspective. Interactive Educational Multimedia, n.4 ( April), pp. 24-39. 

Segelstrbm, F. and Holmlid, S., 2009. Online services and cultural diversity: Applying Hofstede’s framework to predict 
potential issues. In Stauss, B, Brown, S.W., Edvardsson, B. and Johnston, R. (eds) Proceedings of the QUIS 11 - 
Services Conference, Wolfsburg, Germany, pp. 604-612. 

Shostack, G. L., 1982. How to Design a Service, European Journal of Marketing, Vol. 16 No 1, pp.49 - 63. 

Van der Veer G.C. and van Welie M., 2003. DUTCH - Designing for Users and Tasks from Concepts to Handles. In: The 
Handbook of Task Analysis for Human-Computer Interaction. Dan Diaper & Neville Stanton (Eds). Lawrence 
Erlbaum, Chapter 7 pp 155-173. 

Van der Veer, G.C., Consiglio, T. and Benvenuti, L., 2011. Service Design - a structure for learning before teaching. In 
Patrizia Marti, Alessandro Soro, Luciano Gamberini and Sebastiano Bagnara (Eds) Facing Complexity - Adjunct 
Proceedings CHItaly 2011. ACM Digital Library, pp. 144-147. 

Vygotsky, L.S., 1962. Thought and Language. MIT Press, Cambridge, MA. (original work published 1934). 

Williams, K.C and Williams, C., 2011. Five Key Ingredients for Improving Student Motivation. Research in Higher 
Education Journal, August- pp. 1-23. 


132 



lADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2013) 


ISSUES OF LEARNING GAMES: 
FROM VIRTUAL TO REAL 


Thibault Carron\ Philippe Pernelle^ and Stephane Talbot^ 

‘Lip6 Lab Paris 6 & Universite de Savoie 
^Disp Lab, Universite de Lyon 
^Universite de Savoie 


ABSTRACT 

Our research work deals with the development of new learning environments, and we are particularly interested in 
studying the different aspects linked to users’ collaboration in these environments. We believe that Game-based Learning 
can significantly enhance learning. That is why we have developed learning environments grounded on graphical 
representations of a course. These environments allow us to set up experiments with students in our university. The 
emergence of online multiplayer games led us to apply the metaphor of exploring a virtual 3D world, where each student 
embarks on a quest in order to collect knowledge related to a learning activity. In the environment, each part of the world 
represents a place, sometimes a collaborative place, where students are supposed to acquire a particular concept. Learning 
objects, artefacts or collaborative tools may be present in each location and a correct answer to a specific exercise gives a 
key to the students, allowing them to access other activities. 

Although the students appreciate this approach, there is a lack of assessment of know-how-type skills, especially for the 
teacher. Indeed, certain domains present the particularity of exhibiting both theoretical knowledge and practical know- 
how (operations in manufacturing or medicine, for example). For such contexts, the current Learning Games are not 
efficient concerning this second point: a unique and full digitalisation of these objects alone is not sufficient to guarantee 
both effective learning and assessment of the techniques. We consider this as a gamification problem. As a matter of fact, 
in industrial domains, the learning processes are often based on the use of certain objects that are difficult to include in a 
learning game. Moreover, although some games are collaborative, an effective collaborative activity is more effective in 
a real context. Our objective is then to facilitate the transfer by integrating a new object present both in the game and in 
the classroom. 

In this paper, we focus on the way to integrate the use of real objects into the learning game environments. In our 
approach, we develop a digital copy of the object that will help to exhibit the specific know-how that must be evaluated. 
It is then easier during the gamification process to include in the learning scenario certain parts that are relevant to the use 
of such an object and that are achieved in a real context. First, we describe briefly the “Learning Adventure” 
environment: a generic game-based platform. Then, we explain what the new issues of such contexts are, and how we are 
able to setup a collaborative learning session with both virtual and real objects. The third part concerns the application to 
a concrete problem: to identify and manage Nonconformities thanks to a Product Lifecycle Management tool. A real 
experiment has thus been carried out at our university in the PLM domain with the help of a multi-touch tabletop, to 
validate the feasibility of the approach and illustrate our point. 


KEYWORDS 

Serious Game, mixed reality. Product Lifecycle Management, nonconformity, collaborative activity. 


1. INTRODUCTION 

Nowadays, compared to traditional teaching methods, Learning Management Systems (LMS) offer 
functionalities that are recognized as being valuable from different points of view. For instance, students can 
learn at their own speed. These environments also allow the teacher to evaluate specific activities in a 
uniform way. However, although they enable powerful features, they also receive major kinds of criticism 
(lack of awareness, few collaborative or regulation possibilities (Kian-Sam 2002)). Moreover, some students 
tend to consider LMS as unexciting (Prensky, 2000). 
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That is the reason why, observing the emergence and success of online multiplayer games with our 
students, we decided to adopt a learning game approach, by developing our own game environment and by 
using it as a support for some learning sessions. Agreeing with Vygotsky’s school of thought and activity 
theory (Vygotsky, 1934), we consider that the social dimension is crucial for the cognitive processes 
involved in the learning activity. Indeed, we think that the way of acquiring knowledge during a learning 
session is similar to following an adventure in a Role-Playing Game (RPG). The combination of the two 
styles is called MMORPG (Massively Multiplayer Online RPG) and offers a good potential for learning 
(Galameau, 2007) reformulated as MMOLE (Massively Multiplayer Online Learning Environment). 
Nevertheless, although the students appreciate this approach and that Game-based pedagogic tools can 
significantly enhance learning, there is an obvious need for realistic and reliable assessment about students’ 
skills, actions or behaviours, especially for the teacher. Indeed, for certain specific domains, the teacher needs 
to evaluate his/her pedagogical session according to two specific points that are particularly difficult to assess 
via such a learning environment. Certain domains present the particularity of exhibiting both theoretical 
knowledge and practical know-how (operations in manufacturing or medicine, for example). For such 
contexts, the current Learning Games are not efficient concerning this second point: a unique and full 
digitalisation of the objects alone is not sufficient to guarantee both effective learning and assessment of the 
techniques (Schrier, 2006). 

In this article, we will focus on these two points: the know-how and the collaborative behaviour. We will 
first describe the Game-based learning environment that we have developed, and next focus on integrating a 
new object concerning the two sides of the world - real and virtual - in order to set up a mixed reality learning 
game. The main point of this article is how to take into account the new issues due to the mixed reality, and 
in a secondary point how to gather in the same environment information concerning general learning 
concepts and business concepts. Finally, in the last part, we will try to illustrate most of these points by 
means of a concrete example and set out a way via a real experiment to identify and manage non- 
conformities thanks to Product Lifecycle Management in such a learning environment. 


2. LEARNING ADVENTURE ENVIRONMENT 

Learning Adventure is a Game-Based Learning Management System representing a 3D environment where 
the learning session takes place (see figure 1). A particular map (buildings, enterprise areas but also the 
environment with lakes, mountains and hills) is dedicated to a particular learning activity, for a particular 
subject. Each part of the map represents the place where a given (sub-) activity can be performed. 



Figure 1 . An Industrial Map in the Learning Adventure Environment 

The map topology represents the overall scenario of the learning session, i.e. the sequencing between 
activities. There are as many regions as actual activities, and the regions are linked together through paths 
and NPC (Non Playable Character) guards, showing the attainability of an activity from other ones. 
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An example of a scenario seen as a map topology is presented in figure 2. Similar models that link 
pedagogical issues with game elements can be found with a more general point of view in (Amory 1999) and 
more precisely concerning this approach in (Carron 2008). 



Eigure 2. An example of a Scenario Seen as a Map Topology 

As explained previously. Learning Adventure is based on a role-play approach (Baptista 2008). Players 
(students or teachers), possibly represented by their own avatars, can move through the environment, 
performing a sequence of sub-activities in order to acquire knowledge or/and solve problems (here finding 
and identifying non conformities, see figure 2). Activities can be carried out in a personal or collaborative 
way (see (Dillenbourg 1996) for a list of cooperation abilities); one can access knowledge through objects 
available in the world, via help from the teachers, or from work with other students. 

Although such game environments and characteristics are well known to our students, the so-called 
«digital natives», some reminders are always proposed at the beginning of the pedagogical session. This first 
playable part called the Newbie Park allows us to describe the main functionalities, and explain the use of 
certain specific collaborative tools that are present in this game or this particular learning session. Moreover, 
as in many collaborative sessions, this part can be seen as a “warm-up activity” in order to get students’ 
minds into an appropriate “ready to play for learning” state. As regards Learning Games, our experience 
shows that such a first step is crucial, as is a final debriefing phase in order to help cement the knowledge 
acquired (Garris et ak, 2002). 

The environment is generic in the sense that it is not dedicated to a particular teaching domain. With the 
help of a pedagogical engineer, the teacher adapts the environment before the session by setting pre- 
requisites between sub-activities and by providing various resources (documents, videos, quizzes) linked to 
the course. Experiments have been set up for learning English as well as Project Management or Object 
Oriented Concepts in Computer Science. The collaboration takes place in L.A. by constituting groups of 
users. The NPC will give objectives to the members of a group and allow them access to collaborative tools 
such as white boards, file boxes or a “collaborative plan elaborator” similar to a structured discussion forum. 
It is then possible to construct group knowledge with specific tools. Naturally, in order to communicate with 
other players a chat tool is available (bottom left comer in figure 1). 

It is possible for the students to embark on several knowledge quests in parallel. That is why it is 
important for them to be aware of the quests currently active. In order to facilitate navigation inside the game, 
the objects or NPCs to be reached in the active quests are displayed on a compass (upper right corner in 
figure 1). Moreover, one can easily reach one’s user model and consult the current content of one’s bag (see 
bag window open and icons on the right hand part of figure 1). Nevertheless, as explained previously, certain 
specific skills (particular manipulation, know-how) or behaviours (collaboration) are difficult to assess by 
staying only in the virtual world, and underscore the necessity of reintegrating some parts of the game into 
the real world. Prom our point of view, the relevant work is not trivial in nature and raises some new issues 
that have to be taken into account in order to keep the advantages of serious games (immersion, motivation). 
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3. ISSUES OF THE GAMIFICATION PROCESS WITH MIXED REALITY 
GAMES 

In order to illustrate our point, we will take the example that will be the focus of the experiment described 
below. 

In quality management, a nonconformity is a deviation from a specification, a standard, or an 
expectation. Certain specific information systems known as PLM (Product Lifecycle Management) are 
dedicated to providing information about such defects. It is quite difficult to identify, via a serious game, 
such skills that generally impose debates between the students/players featuring descriptions of the problem. 

For example, our proposition was to create both a specific tool and an object able to exhibit such skills. 
For our example, it was important for a student to be able to present to the other ones and observe with them 
the possibly defective objects: we chose to create a multi-touch tabletop in order to provide a collaborative 
tool for that purpose. Nevertheless, depending on the context, many other interface objects could be 
envisaged (Feedback/Data gloves (haptic devices), 3D goggles, Kinect, physical objects (“tangible things”), 
QR Code, NFC Tag, etc.). 



Figure 3. Students Debating About Nonconformity around the Table and the Representation of the Corresponding Scene 

in the Learning Game 


The problem of the gamification process (i.e. the transformation of common business processes into one 
or several game scenarios) is already well identified, but concerning MRLG (Mixed Reality Learning 
Games), some brand new problems have appeared. 

First of all, to remain coherent, it is important to find a way to pass “naturally” from the virtual world to 
the real one. Our idea was to use a specific object that existed in both worlds. This object would be used to 
support this transition smoothly (like a transitional object in the psychological domain). Another good side 
effect of this object is to help the user to reinvest their acquired knowledge more easily outside the virtual 
game. 

A second problem concerns the access to this object: as we are working in a collaborative session, several 
students may require access to this object at the same time. When this object represents a (group) exclusive 
(or non-shareable) resource, the learning scenario has to be adapted to such a configuration in order to avoid 
any delay. As a matter of fact, any delay would result in instant disengagement of the students. 

Moreover, it is now well known that good collaboration is effectively achieved with a group of 3-5 
persons at most. We have then to ensure that the players of the same groups will progress together at the 
same speed and will be synchronized thanks to the scenario evolution. 

A Serious Game concerning change management has already been proposed as a solution for specific 
training in matters of industrial problems. The purpose here is to be more general and go further concerning 
the gamification process by including mixed reality in such learning games. Mixed reality has allowed us to 
take into account new needs, but it also means that some new problems have to be addressed. 

As seen in table 1 , for each problem relative to mixed reality that we have detected, we have proposed a 
solution. 
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Table 1. Examples of Mixed Reality Issues 


Domain 

issues 

Virtual World 

Real World 

Pedagogy 

Concrete assessment 

(know-how, manipulation, 
gestures) with a specific device. 

Virtual representation of 
the specific device 

(no evatuation) 

Real Object/ Specific 
Device 

(effective work) 

Entertainment 

Keep the learner immersion 

Coherent Story leading to 
the specific object 

Integration of clues 
and challenges that 
remind us that we are 
in a game. 

Collaboration 

(intra-group) 

Synchronize students in the 
same groups 

Propose some optional 
mini-quests 

Small groups with an 
active role for each 
one. 

Collaboration 

(extra-group) 

Synchronize groups of students 
(no wait for access to a group 
exclusive resource) 

Change the order of the 
quests. 

Access to the real 
world 

Content 

Assessment of content 

Indicators thanks to traces 

Camera or results 
reintegrated into the 
game via digital input. 

Scenarization 

Retain a complete game 
feeling. 

Beginning and end in the 
game. 

Small moments in the 
real world. 


In fact, the main points in the table above are not the solutions that are proposed for illustration but rather 
the issues. These solutions are not exhaustive, and must be adapted to perfectly fit with the context. 
Nevertheless, these issues have to be resolved because setting up such experiments is very costly and 
generally cannot be easily done again. Indeed, logistical problems increase with collaboration and mixed 
reality needs. It is important to anticipate these problems and keep motivation and immersion at a high level. 

These considerations have permitted us to prepare a serious game in mixed reality that has been 
experimented in a real context. The following section describes the content of this scenario. 


4. SCENARIZATION AND APPLICATION TO NON-CONFORMITY VIA 
PRODUCT LIFECYCLE MANAGEMENT 

The implementation of a PLM system significantly alters the organization of the company, particularly in the 
context of SMEs (Small and medium-sized enterprises). Resistance to change (individual and collective) 
appears naturally during the start-up of this type of system (Kadiri et al. 2009). Our learning environment is 
defined for supporting change management in industrial enterprises and to help companies to effectively 
support their staff in adapting to the changes brought about by redesigning their information systems. The 
challenge is now to train the employees for them to have the ability to detect and manage nonconformities 
with such an attractive environment. The relevance of this serious game is twofold; the use of video game 
techniques to teach the use of PLM to detect nonconformities and evaluate the contribution of mixed reality 
in assessing particular skills concerning the identification of nonconformities. 

Concerning the assessment, all tools are equipped with specific probes in order to obtain information 
about the learning progression of the students (Marty et al., 2012), even for the specific device where the 
results are reinvested in the game. For example, all the actions carried out through the interface are collected 
in the game. We are thus able to know who has made an incorrect manipulation of the business tool or the 
business process, provided a positive or a negative comment, asked for an explanation, etc. For very 
particular assessment (that is not numerically observable), we also use camera and real observers present in 
the room. Generally, the evaluation of the use and ownership of content by a user is needed. 

For this collaborative scenario (see figure 2), we have defined two important activities (Al and A2) that 
are the main quest in sequence and three sub-activities (A 1.1, A 2.1, A 2.2) that are optional (for 
synchronization purposes: see table 1). We can add AO for a starting activity and A4 for an ending one. 

The scenario seems basic, but the sequence was in reality not fixed, and three ways of achieving it were 
expected. The narration proposed by the NPC changed with each group in order to enable access to the multi- 
touch tabletop at different moments of the scenario and avoid any delay. 

One of the difficulties of such games is the definition of industrial scenarios. The aim of the modeled 
scenarios is to understand the use of PLM in identifying and managing nonconformities. The table below 
shows the steps concerning the main features that we want to evaluate. 
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Table 2. Steps of Usage Scenarios 


Steps of the scenario 

Example 

Observe the environment 

A battery silo is different from the others 

Identify five possible NCs 

This saber is too short 

Understand the use of PLM as a tool for 
collaboration 

Have a look at the specific orders that could 
explain something unusual 

Collaborative Exchange 

Identify formally three NCs 

Validation 

Input/submit the proposition to the chief 


This process is carried out in the Serious Game with a connected PLM system as well as access to a 
collaborative platform in order to reach some additional learning resources (videos, tutorials). The actions 
that can be performed in the game are the following: discussion with colleagues and watching a training 
presentation, collecting information (taking pictures) in order to debate about nonconformities and visualize 
these pictures on the multi-touch tabletop, filling in the nonconformity form, retrieval of information from 
the archives, etc. The results are sent via the PLM tool to the teacher who validates or invalidates the choices. 

This collaborative scenario has been experimented several times in an ecological context, as we will 
describe in the following section. 


5. EXPERIMENT DESCRIPTION AND RESULTS 

The experiment in real conditions concerning Nonconformities and Product Lifecycle Management has been 
realised in order to validate our approach at the Lyon Institute of Technology with master’s students. 

5.1 Conditions and Methodology of the Experiment 

This experiment was carried out in 2013 at the University of Lyon with co-located settings. During the 
experiment, a group of 12 students (three groups of four persons simultaneously) with their teacher were 
present successively in a classroom equipped with 12 computers. Concerning the social presence perception 
(see (de Kort, 2008)), players were oriented away from each other, limiting mutual eye contact, natural 
reciprocation of approach or avoidance cues and mirroring. The students were between 24 and 50 years old 
(PLM Master’s) and for the most part familiar with computer use. Each student accessed the virtual 
environment through his/her workstation, and had a personal (adapted) view on the world. These students 
used the environment for approximately two hours. They were explicitly allowed to communicate through 
the chat tool provided with the system and were warned that they would be observed concerning the use of 
the system. At least two observers were present in the classroom. The students were free to refuse this 
observation (the same practical work was available outside the learning environment), but everyone agreed to 
follow the proposed protocol. Finally, the first part of the session consists of a fifteen-minute introductory 
section called the «newbie park», used in order to present and explain step by step the basic functionalities of 
the game (collect bag, quest book, skill book). 

The same experiment has already been carried out four times (48 students) with similar results that we 
will explain in paragraph 5.3. 

5.2 Support for Pedagogical Objectives 

The pedagogical objectives explained in the preceding section are supported by a story that guides the 
knowledge quest with the help of metaphors. For motivation purposes, we integrate the scenario into a 
futuristic world. Indeed, the challenge is encouraged through NPCs which propose a coherent contest. 
Immersion is reinforced when the users' actions have a direct impact on the objects of the world. Finally, the 
teacher was present in the game via an avatar: it was possible to chat with him, to ask for help for example. 
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5.3 Evaluation and Results 

Concerning this experiment, two means of evaluation were chosen; 

• Quantitative, thanks to collaborative indicators elaborated with traces left by the users when 

collaborating, 

• Qualitative, with live questions at the end of the session and explicit feedback from the 

teacher/PLM expert. 

At the end of the experiment, the students were asked to answer questions to give feedback about their 
feelings concerning their work session. The questions (ranking and open-ended ones) evaluated aspects 
relating to several parts of the learning game (pedagogical content, business concepts, affordance of the 
objects or tools, scenario-story, immersion, collaborative activities and specifically collaborative tools, and 
user model evolution). The final question let the students propose improvements concerning weak and strong 
points of the game. For example, they found that other sub-quests should be proposed, and some people were 
very interested in exploring the map (and found jokes or amusing allusions). This classification of gamers is 
well-known (Yee, 2007). 

The initial objective concerning nonconformity identification was achieved. The students were 
unanimous in preferring to work with this environment rather than doing conventional practical work on 
workstations, and more generally were very enthusiastic about this kind of experiment. 

From the teacher’ s point of view and in the light of previous experiments, it was naturally mandatory to 
have specific tools supporting him in the monitoring task with the help of an updated user model for each 
student. As a matter of fact, the teacher is present in the game to validate propositions of nonconformity, but 
also to give access (/or deny access) to the right students to the “debate room with the real tabletop”. 

The results were satisfying but the setting-up of such tools and experiments is, as always, extremely time- 
consuming for the teacher - indeed, even more so than usual. We will see later whether all these facilities 
may be reused easily in other domains. Other experiments with the same environment but applied to a 
different domain (electronic teaching with arduino electronic shields) are currently planned with a larger 
public (60 students) and in another institute of technology. 


6. CONCLUSION 

In this article, we have illustrated a way of integrating real tools into a collaborative session of a learning 
game in order to be able to assess particular skills or behaviors. Thanks to information found both in the 
game during the pedagogical session and through the use of a professional business tool, and also with the 
use of a multi-touch tabletop, we validated this approach with an experiment with our students concerning 
the management of nonconformity with Product Lifecycle Management Systems, the said experiment being 
carried out in the Learning Adventure environment. This environment is collaborative, multiplayer and fully 
observable thanks to traces left during the game. These traces allow us to elaborate collaborative indicators. 
Moreover, through the feedback collected from these experiments, we are able to obtain new factual 
indicators of collaboration by exploiting the traces left by the users. 

As stated previously, this environment is still being developed and will be aimed at proposing both 
specific collaborative tools and facilities to help the teacher to regulate a learning session. Moreover in 
future, we would like to see how teamwork may be self-regulated thanks to specific functionalities of 
collaborative tools. We have shown an example based on several players interacting and communicating in 
the same environment, but with the use of real teams and roles, we may also imagine self-regulation, team- 
regulation or auto regulation by the use of specific rules, as can be the case in Artificial Intelligence. Self- 
regulation, co-regulation and socially-shared regulation are precisely described in (Hadwin 2010). 

Naturally, certain drawbacks persist: we must recognize that it is very difficult for the teacher to be 
present in the game, help the students and regulate the session, even with these specific tools. For the 
moment, the indicators that are present are not very well-integrated, because they are very time-consuming to 
develop. We currently think that we can develop some specific generic indicators dedicated only to classical 
fields of a domain. 

An interesting perspective could be to develop and propose directly within the indicators some basic 
regulation actions such as “play specific PLM video”, “propose new (adapted) content or new (adapted) 
scenario” or “enable/disable such facility/ies for this student”. 
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Finally, in another project called SLI (Serious Lab for Innovation), again supported by the French 
Ministry for the Economy, Industry and Employment, we also applied this work to another domain: 
Innovation in industry. Naturally, another scenario was imagined, developed and proposed for that purpose. 
Some qualitative results concerning concepts and methodology about innovation are still under processing. 

Erom a more general point of view, “conducting change” is a key issue both for industrialists and 
students, who will experience such a challenge later on. A generic serious games approach will help us to 
link the gap between education courses (a theoretical approach thanks to pedagogical resources left in the 
game) and concrete experimentation in a professional context thanks to the integration of a real business tool 
(access to interfaces via web services). 
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ABSTRACT 

Among the unique affordances of digital simulations are changes in the possibilities for targets as well as the methods of 
assessment, most significantly, toward integration of thinking with action, embedding of tasks-as-performance of 
knowledge-in-action, and unobtrusive observational methods. This paper raises and briefly defines key data challenges of 
assessing learning in a complex domain of performance within a digital simulation, which at the atomistic level include 
time and event segmentation, cyclic dynamics, multicausality, intersectionality, and nonlinearity. At the summary level, 
the key challenge is model building. An example of a simulation designed to develop teachers - simSchool - is integrated 
with an adaptive content delivery and analytics database - Leverage - which grounds the discussion. 


KEYWORDS 

Learning analytics, simulation, digital game-based learning, adaptive learning 


1. INTRODUCTION 

Assessing learning in a simulation requires a formalization of familiar everyday reasoning that assumes if 
someone is observed saying or doing something, the observation can be used to infer what they know and 
know how to do (Pellegrino, Chudowsky, & Glaser, 2001). The inferences require new methods of analysis 
because the performance space of simulations is considerably more complex than a traditional psychological 
test or measurement (Aldrich, 2004; Behrens, Frezzo, Mislevy, Kroopnick, & Wise, 2008; Rupp, Gushta, 
Mislevy, & Shaffer, 2010). For example, users acting within a digital environment contend with elements 
such as the interface tools, and purposes that are inherent in both the design of the environment and emergent 
in its interaction with users. Players also contend with anchored and evolving interaction rules, other players, 
and their private mental models of the environment as they traverse the available landscape of possibilities of 
thought and action. Learning what those users know and can do based on their actions as well as the artifacts 
they create in the digital environment involves three phases of dynamic assessment (Quellmalz et ak, 2012); 
gathering data, applying criteria to make inferences and claims, and undertaking adaptive interactions with 
the user such as reporting results, offering new digital experiences, or ending the interaction. 

This paper briefly defines key data challenges we have been facing and addressing in order to assess the 
complex higher order skill of classroom teaching based on evidence provided by simSchool as captured, 
analyzed and reported by Leverage. simSchool is a digital flight simulator for teachers, which provides a 
performance and assessment platform for the development of teaching skills. Leverage is a user analytics 
application for data mining interactions in digital environments; these applications work together to form a 
digital media-learning environment with embedded assessments of higher order knowledge and skills. 
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2. LEVERAGING A SIMULATION 

Across a variety of settings, artificial intelligence and user analytics engines in simulations have been found 
useful for representing and developing higher order skills such as leadership, responsibility and time 
management; skills that are displayed when users interact with, influence others and make decisions. 
These settings include human resource departments, medical training programs, professional development of 
counsellors, military leadership training, and teacher education programs - anywhere that experts are 
interested in developing new supervisors, team leaders, and teachers (Aldrich, 2004; Gibson, Aldrich, & 
Prensky, 2007; Prensky, 2001). 

In simSchool (www.simschool.org) a user plays the role of a teacher while computer resources play the 
role of students. An artificial intelligence engine handles user interactions guided by models of teaching and 
learning; and a user analytics engine. Leverage (www.pr-sol.com), handles the delivery of digital media, the 
administration of groups of users, and the assessment of learning, including capturing, analyzing and 
reporting on data. 

The data produced during a session has a dual role: it is used to influence the education of, as well as to 
make inferences about what the user knows and can do as a teacher, that is, within the epistemic frame (D. 
Shaffer, 2007) of the profession. The dual role and epistemic positioning illustrates how a simulation can 
simultaneously fulfil roles in the assessment FOR, OF and AS learning (Bennett, 2010). As a player makes 
choices in simSchool, a digital trail collected by Leverage provides evidence of the player’s teaching 
expertise that is revealed in how and to what extent the simstudents learn as well as how the user manages 
available resources including instructional moves and student communications during the simulation. 

2.1 Simulations as Complex Systems 

The details of how simSchool works - how the simulated students respond to tasks and teacher talk - have 
been detailed elsewhere (Christensen, Tyler-Wood, Knezek, & Gibson, 2011; Zibit & Gibson, 2005; Zibit, 
Gibson, & Halverson, 2006) and is only briefly outlined here in order to focus on the data challenges of 
embedded and automated assessment. In brief, simSchool uses a dynamic modeling approach in which the 
user is a teacher who is an independent actor that chooses tasks and talking interactions, which in turn act as 
attractors for the simstudents. The artificial intelligence driving each simulated student is a hill-climbing 
algorithm; each student will attempt to reach equilibrium by attaining the goals of a given task if the task and 
setting do not impose too many barriers and the system is not perturbed by any other user actions. The time it 
takes simstudents to reach equilibrium with a task is determined by how their personality variables (physical, 
emotional and cognitive variables) interact with the requirements of the tasks and the teacher’s talking 
choices. 

The simSchool game mechanic ensures that the difference between any starting condition and any current 
or ending condition of the game is a result of the decisions made by the player. If a simstudent has learned or 
failed to learn, it is directly traceable to the user’s decisions. While it might seem to oversimplify complex 
teaching practices, this arrangement actually allows a wide variety of performances by simSchool users, with 
potential for a number of inferences that can be made based on the digital record as well as by pre and post 
assessments and concurrent observations of the users. 

2.2 Leveraging Data 

Leverage software by Pragmatic Solutions, fully integrates with simulations such as simSchool and provides 
a scalable server application and database backend that is an infrastructure for scaling the application, 
organizing all users and groups, and creating extensive learner analytics. Server functions that summarize 
data are based on atomistic and summary events, which consist of a value associated with action(s) or states 
and their time-based contexts. Events identify a discrete user or application activity and exist at a variety of 
scales (atomic to summary levels) of data creation and collection, recognition of which feeds the assessment 
process as well as the adaptive digital media learning experience of the simulation. In Figure 1, we show an 
example of the time sensitive evolution of a cluster of variables in a simSchool teaching situation. 
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Eigure 1. Representations of time-based events and a graph from an overview spreadsheet summary in simSchool display 
how variables in the simulation change over time in relationship to user actions. Changes in tasks made by the user are 
represented by colored blocks, time flows from left to right. Changes in simstudent internal states caused by the tasks are 
represented as points of data forming lines that change over time, displaying whether and to what extent the simstudent 

adapts and learns. 


3. ATOMISTIC DATA CHALLENGES 

Data challenges at the atomistic level are numerous, including time and event segmentation, cyclic dynamics, 
multicausality, intersectionality, and nonlinearity, which we will discuss in this section. Advanced 
multivariate methods involving serial and canonical correlations for multiple variables combined with 
automated network analysis methods provide some solutions and methods for these data challenges. We 
admit here that we have much yet to learn to integrate and place these methods into an automated 
computational framework with both inductive and deductive capabilities so that near-real time feedback can 
be provided to users of digital media learning environments and useful summaries can be created analyzing 
what users know and are able to do. 

3.1 Time and Event Segmentation 

The time segmentation problem is illustrated by the fact that since simSchool data captures performance of 
knowledge-in-action, how should we represent and analyze knowledge in vivol That is, we’d like to be able 
to say things about what the user knows and can do without killing or masking critical performance 
information that evolves over time. One possibility we’ve explored is to provide time-based representations 
for human and machine analysis and to develop time-sensitive automated analytic methods for making 
inferences. 
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In Figure 1, approximately 20 data captures are taken within a span of about 3 minutes. Four user choices 
of task are evident, as are the impacts of each of those decisions on the learning of one simulated student, 
represented by changes in 6 dimensions as the agent adapts to the different task requirements. 

If we conduct a summative assessment at the end of the first task “do a team worksheet,” we’d have to 
conclude that the user’s choice of task caused important variables to decline in the agent, most importantly, 
academic performance. But if we wait until the end of the next task “take notes” the agent begins to recover 
academically. This illustrates one of the most basic problems of time-sensitive data analysis: since systems 
evolve over time, how much time do we need in an analysis and when is the best time to stop gathering and 
start making sense of the situation? Is there a form of continuous interpretation that should be employed, and 
should it use all the data or a particular window of time? This first problem includes determining what some 
have called slices, episodes or segments (Choi, Rupp, Gushta, & Sweet, 2010; Rupp et al., 2010; D. W. 
Shaffer et al., 2009) and it is not clear yet how to make differently sized slices commensurate with each other 
when the timing aspects are critical to the analysis. 

3.2 Cyclic Dynamics 

A second problem closely related to time, is the partially closed loop or cyclic dynamics problem of 
causality. Complex systems, such as a user making choices and interacting with a digital artifact that in turn 
responds to those choices over time, have loops or cycles of causes (e.g. chicken and egg dilemmas). By 
“partially closed loop”, we mean that one thing is both a cause and an effect of something else, which might 
in turn be both the cause and effect of the first thing, in a loop of ongoing relationships. Stopping the cycle is 
an arbitrary moment in the co-evolving causal network (e.g. imagine measuring the state of a room, cooling 
engine and thermostat on a day that begins cool, warms up and cools down again). In addition to the intrinsic 
self-reinforcing loops, external drivers may also be varying, raising the possibility that other things can also 
be both causes and effects connected to the loop. The practice of assuming that a cause precedes an effect - 
a mainstay of linear causality - might be unwarranted. If we take a snapshot at any point in time (e.g. any 
summative assessment) then we catch the system at some point in its loop, but we don't get the full loop into 
that one picture. For cyclic processes, what is the minimum number of measures per cycle and the 
characteristics of those measures that will produce a particular level of accuracy in the representation and 
analysis? We need methods that include effects as causes as the complexity of interactions evolve and that 
represent the differential phases of a loop of relationships when the loop is acting as a cyclical causal factor 
in a dynamic situation. 

An example of data from a simSchool simulation illustrates these and other challenges in building user 
analytics and interpreting the results of actions in a simulation (Table 1) during approximately 5 minutes of a 
user’s performance. The first ten columns represent states of a single agent (a simulated student), with 5 
variables for five psychological components (known as OCEAN), 1 variable for academic position, 2 
computed variables that aggregate a running average from OCEAN (Power = Eh-Ch-O and Affiliation = 
Ah-N), 1 variable that relates power and affiliation to an altitudinal position on the Interpersonal Circumplex 
(Hofstee, de Raad, & Goldberg, 1992; Plutchik & Conte, 1997), 1 variable holding the pose or body position 
of the student at the classroom desk, and 3 variables that hold the intention of the user when talking to the 
student, in terms of the content of the talk (e.g. is the comment about student behavior or academic 
performance), the type of statement (e.g. is the comment an assertion, observation or question) and the 
altitudinal stance (e.g. one of 16 positions on the Interpersonal Circumplex see (Zibit & Gibson, 2005). 
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Table 1. Data from five minutes of a simSchool simulation. Trajectories for each field are estimated with data captured 
every ten seconds; 32 records were captured in 5.33 minutes of real time. 


row 

E 

A 

C 

N 

O 

Ac a 

Power 

Affli 

C 

P 

Re 

Ty 

An 1 

1 

-0.506 

-0.197 

-0.451 

-0.451 

0.910 

-0.036 

-0.016 

-0.324 

4 

E 




2 

-0.509 

-0.269 

-0.428 

-0.428 

0.873 

-0.054 

-0.021 

-0.348 

4 

E 




3 

-0.511 

-0.329 

-0.405 

-0.405 

0.840 

-0.071 

-0.025 

-0.367 

4 

E 




4 

-0.514 

-0.378 

-0.382 

-0.382 

0.811 

-0.088 

-0.028 

-0.380 

4 

E 




5 

-0.517 

-0.418 

-0.360 

-0.360 

0.786 

-0.105 

-0.030 

-0.389 

4 

E 




6 

-0.519 

-0.450 

-0.339 

-0.339 

0.763 

-0.120 

-0.031 

-0.395 

4 

E 




7 

-0.522 

-0.477 

-0.318 

-0.318 

0.744 

-0.135 

-0.032 

-0.398 

4 

E 




8 

-0.524 

-0.499 

-0.297 

-0.297 

0.727 

-0.149 

-0.031 

-0.398 

4 

E 




9 

-0.526 

-0.518 

-0.277 

-0.277 

0.711 

-0.162 

-0.031 

-0.397 

4 

E 




10 

-0.528 

-0.532 

-0.257 

-0.257 

0.698 

-0.174 

-0.029 

-0.395 

4 

E 




11 

-0.531 

-0.545 

-0.238 

-0.238 

0.686 

-0.185 

-0.028 

-0.391 

4 

E 




12 

-0.381 

0.195 

-0.067 

0.531 

0.828 

-0.196 

-0.127 

-0.363 

5 

E 

b 

in 

3 

13 

-0.387 

0.052 

-0.052 

0.532 

0.801 

-0.202 

-0.120 

-0.292 

5 

E 




14 

-0.394 

-0.065 

-0.038 

0.534 

0.777 

-0.205 

-0.115 

-0.234 

5 

E 




15 

-0.400 

-0.161 

-0.023 

0.535 

0.755 

-0.208 

-0.111 

-0.187 

5 

E 




16 

-0.406 

-0.240 

-0.009 

0.537 

0.737 

-0.212 

-0.107 

-0.148 

6 

E 




17 

0.210 

-0.305 

0.626 

0.538 

1.342 

-0.218 

-0.726 

-0.116 

8 

E 

b 

ob 

0 

18 

0.185 

-0.358 

0.625 

0.539 

1.253 

-0.238 

-0.688 

-0.091 

8 

E 




19 

0.162 

-0.402 

0.624 

0.541 

1.174 

-0.257 

-0.654 

-0.070 

8 

E 




20 

0.139 

-0.437 

0.624 

0.542 

1.105 

-0.275 

-0.623 

-0.052 

8 

E 




21 

0.117 

-0.467 

0.623 

0.543 

1.045 

-0.292 

-0.595 

-0.038 

8 

E 




22 

0.095 

-0.491 

0.623 

0.545 

0.991 

-0.308 

-0.570 

-0.027 

8 

E 




23 

0.075 

-0.510 

0.622 

0.546 

0.944 

-0.323 

-0.547 

-0.018 

8 

E 




24 

0.054 

-0.526 

0.622 

0.547 

0.903 

-0.337 

-0.526 

-0.010 

8 

E 




25 

0.207 

-1.290 

0.793 

-0.202 

1.039 

-0.350 

-0.680 

0.746 

2 

D 

b 

as 

13 

26 

0.182 

-1.166 

0.789 

-0.184 

0.986 

-0.378 

-0.653 

0.675 

2 

D 




27 

0.159 

-1.064 

0.785 

-0.166 

0.940 

-0.400 

-0.628 

0.615 

2 

D 




28 

0.136 

-0.980 

0.781 

-0.149 

0.899 

-0.418 

-0.605 

0.565 

2 

D 




29 

0.114 

-0.912 

0.777 

-0.132 

0.863 

-0.433 

-0.585 

0.522 

2 

D 




30 

0.093 

-0.856 

0.773 

-0.116 

0.832 

-0.445 

-0.566 

0.486 

2 

D 




31 

-0.128 

-0.241 

0.569 

0.469 

0.604 

-0.454 

-0.348 

-0.114 

7 

E 

a 

as 

5 

32 

-0.142 

-0.306 

0.569 

0.472 

0.603 

-0.453 

-0.344 

-0.083 

7 

E 





Note: Eield labels = row, extroversion, agreeableness, conscientiousness, neuroticism, openness, academic, power, 
affiliation, circumplex, talk about, talk type, talk attitude 


For simplicity, an additional persistent set of variables that hold the constant targets of the task chosen by 
the user is not shown in Table 1. Those task data influence all rows in this series as an attractor, the goals 
toward which the OCEAN and academic variables are striving. The actual values of most variables is in 
floating point precision, but has been formatted to 3 places for convenience. A simSchool session might have 
from 1 to 20 agents each on their own multidimensional trajectory. Updating each row is a set of 
simultaneous equations that integrate the task goals, the previous states and interruptions such as the teacher 
talking during the task (e.g. rows 12, 17, 25, 31). The equations can be independent of each other during each 
time slice, but in fact are deterministically related to each other from record to record, so there is an inherent 
circularity that is only broken by the fact that time moves forward creating the next whole -system state. The 
user is the independent variable in an ongoing experiment during each simulation, determining which task 
the user chose and why, and what the user intended by talking to a particular student during the task. These 
parts of the data are crucial to an analysis of the user, particularly how to think about the meaning of those 
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choices and the resulting data that is generated in order to give feedback to the user both during the 
simulation and afterward. 

3.3 Multicausality 

Choosing a time frame for averaging, finding minimums and maximums and performing other calculations, 
is challenging because significant causes may be present at multiple time scales (e.g. the task is present 
during this entire sequence, but teacher talk events are point-like pulses, and the ongoing variable updates 
each have their own time frames for calculations). We refer to this as a third data challenge - multicausality - 
which traditional statistical models sometimes approach via multivariate methods. Multivariate correlations 
are expressed in terms of strengths summed over some period of time or group of data. However, at some 
scales, short-range dynamics are crucial to long-range causes, but would be obscured by summation over 
time. 

How should we represent and analyze multivariate relationships that are changing over time (perhaps 
rapidly), without oversimplifying them to inert quantities, when their impacts on the system are subtle and 
time-sensitive? For example, a cause may be building for some time before it exerts its influence on the 
system. One potential solution we have explored is to capture the network statistics of many performances 
and use inductively evolved rule sets of a network of relations as a foundation for near-real time assessments 
relating a current performance to that network. 

3.4 Intersectionality 

A fourth problem is intersectionality at multiple scales. Intersectionality is a form of multicausality in which 
influences from diverse scales of space and time arrive at a particular moment in time and space to cause a 
joint effect. This problem implies that we need methods appropriate to represent and analyze dynamic 
geospatial distributions as spaces with probabilities for causing and responding to impacts. Differing amounts 
and types of intersectionality vary by both the scale (e.g. whether the nexus of causes is at the micro, meso or 
macroscopic level of the system) and also by positionality in the network of factors. For example, in Figure 
1, relationships at the horizontal plane that are the historical impacts of task 1, are as important at the time 
task 2 begins, as are the hierarchical relationships of the task 2 requirements. Which exact mixture is 
determining the evolving context of task 2 and how do these relate to the user’s intentions, actions and 
artifacts? 

3.5 Nonlinearity 

A fifth data challenge is nonlinearity. In addition to familiar relationships expressible by nonlinear functions 
(e.g. exponential growth), complex systems generally involve multiple interacting relationships expressible 
by partial differential equations. What are the most appropriate mathematical ideas needed? How much of the 
arsenal of existing statistical analysis do we have to abandon and which tools and methods can we leverage 
as we undertake an analysis that is cognizant of the challenges while also remaining relevant to the domain 
field? We have had success in using symbolic regression (Schmidt & Lipson, 2009), an application of genetic 
algorithms to the discovery of dynamic patterns in complex data. 


4. SUMMARY DATA CHALLENGE = MODEL BUILDING 

One level up from atomistic time-based events are time-independent summaries, which can function as time- 
dependent atoms for larger and larger summaries, giving both hierarchical organization and time-series 
power as a representation and analysis system. Hierarchical temporal features are involved in human memory 
and analysis skills (Hawkins & Blakeslee, 2004), which if understood more clearly, may give future 
automated assessments the ability to think about and process complex human performance information in 
complex ways. To do so, the system of summaries will most likely be organized by a conceptual framework 
that creates chains of evidence from performance information to intermediate and higher levels of 
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representation, some of which are used to report on performance and others as a basis for adapting the digital 
learning experience. Advanced filtering options and data visualization and analysis allow both human and 
machine users to dissect and use summary data. 

Through the Leverage methodology, the cycles of data collection, reduction, analysis, and reporting occur 
simultaneously and continuously during user interactions, and in near -real time, facilitating timely, authentic 
information about user performance. Assessments are created using a scripting language that accesses objects 
such as events and their attributes, player attributes and a subset of summaries called queries. Each 
assessment rule is triggered by an event and processed in order of priority assigned by the creator of the rule. 
Many rules can be processed from a single event, providing an operational platform for subsumption 
architectures (Brooks, 1986, 1999) and quasi-homomorphisms (Holland, 1995; Holland, Holyoak, Nisbett, 
Thagard, 1986) for decisions. Priorities help in processing rules in an order that may contain dependencies to 
other rules. The simulation can thus report activity concerning what the user interface allows as well as the 
internal states of the machine that result from either user interactions or the inherent and emergent behaviors 
of the system’s algorithms. 

At the administrative backend of Leverage is an inductive model builder that creates a representation of 
the aggregated user paths in the network. Assessing proficiency in reaching a goal, as well as the extent and 
contributions of the constituent direct and indirect influences of actions leading to a goal, lies at the root of 
the model-building capability. Initially, assessments provide a mechanism to quantify abstract behaviors in 
the digital space (e.g. integrity or honor in a military simulation, or skill in differentiating instruction and 
understanding the psychology of learners in simSchool) and to pose hypotheses. Leverage then builds a 
representation of the aggregated experience of users of the application in the form of an attributes tracking 
model that reports on the statistical properties of the network. 

An example in Figure 2 shows an assessment determination in the online game and simulation 
“America’s Army,” specifically modelling the Army’s Every Soldier a Sensor developmental initiative. 
Identifying randomly placed target objects in battle impacts the simulation’s scoring using the Army’s core 
value system. Correctly identifying objects involves updating someone’s record of “Honor,” “Duty” and 
triggered by the situation labeled “es20bjectReported.” An assessment developer can see that Honor is 
updated 14.4% of the time when the event es20bjectReported occurs, whereas Duty is impacted 40% of the 
time. On the other end of the influence line between es20bjectReported and the two core values. Duty and 
Honor, the percentage contribution of es20bjectReported toward their updates is so small that a zero is 
reported due to the fact that the simulation is so robust that many other events and assessments contribute to 
Duty and Honor, dwarfing the impact of es2 events. Generally, this particular Leverage analytic tool models 
the relationship between user action (events), rule-driven assessments (queries) and a digital representation of 
a target learning behaviour (attribute). 
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Report Navigation 


Active Game ES2 Prototype 





Figure 2. Inductive mapping of the network formed by user paths creates a first-order understanding of the knowledge of 
users who are performing in the digital space. In this screen, a user’s current state of “Honor” is updated 14.4% of the 

time when an event “es20bjectReported” is reported. 

Network diagraphs (Albert & Albert-Laszio, 2002; Sporns, 2011) such Figure 2 capture important 
weights among resources in the digital learning environment, and represent states of knowledge of the 
network. 

The states can be used in several ways: as benchmarks for performance, for predictions of behavior and 
performance, and to trigger adaptive responses to guide or tutor the player via rewards and consequences of 
actions. The states can be stored and recalled in sequences and in relationship to concurrent slices of the user 
population to computationally represent a chain of evidence that links what a single user does in the 
application, with what everyone does, or what experts do, or what experts want people to do. The network 
viewpoint is thus useful for analyzing a user’s performance within the social and cultural contexts of 
learning, teaching, & educational systems (Gibson, 2006). 


5. SUMMARY 

This paper briefly presented some of the data challenges and analysis approaches in the dynamic 
performance environment of a digital simulation. At the atomistic level of performance events, the challenges 
include time and event segmentation, cyclic dynamics, multicausality, intersectionality, and nonlinearity. At 
the summary level, the key challenge is model building. To ground the discussion, the paper uses an example 
of a simulation designed to develop teachers - simSchool - that is integrated with an adaptive content 
delivery and analytics database - Leverage, and briefly outlines how these two applications work together to 
solve the data challenges. 
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ABSTRACT 

How can student self-assessment be used as a tool and become beneficial for both lecturers and students? We used a 
simple self-assessment tool for pre- and post-testing on a first-semester engineering course. The students graded their 
knowledge on human-computer interaction based on their ability to understand and explain specific concepts. The 
assessment tool was inspired by the SOLO-model. 

The assessment tool promoted practice reflections on the academic concepts. In the pre-test the students became aware of 
specific academic expectations in the course and they got to grade the gap between their own current knowledge and 
what would be expected at the end of the course. The lecturer could right from the beginning sharpen the academic 
semester plan based on the assessment results. 

The post-test could be used by the students as a tool for grading their knowledge in preparation for the final exam. From 
the lecturer’s perspective the post assessment was useful in optimising the course for the next throughput. 
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1. INTRODUCTION 

How can student self-assessment be used as a tool and become beneficial for both lecturers and students? 
From an educational perspective, pretesting students is an important part of tailoring a course to fit with the 
students’ prerequisites. When evaluating the students’ learning at the end of a course it makes sense to relate 
that to their prerequisites in order to measure how much the curriculum of the course has affected their 
learning. Making the students self-assess their knowledge level at the beginning and the end of a course is 
one approach at making the students actively reflect upon their own learning. 

Other resent research has investigated pre- and post-tests of students’ self-assessments (Schiekirka et ah, 
2013), which is based on self-assessment of specific learning objectives related to factual knowledge. They 
conclude that their tool is easy to implement and assists teachers in identifying strengths and weaknesses on 
the level of specific learning objectives for a particular course. 

Within our F‘ semester human-computer interaction (HCl) course, we wanted to promote our students’ 
HCl competences and analytic competences. We wanted the students to create prototypes, be reflective and 
articulate their design process. We believe that dialog based on the academic theory and their design 
experiences reinforces the learning process. This is theoretically supported by Schon (2001) and Bateson 
( 2000 ). 

Furthermore, we wanted to make the students aware of the main topics in the course before we got 
started. We wanted them to take in and reflect on what they were going to learn this semester. This was done 
through a pre questionnaire. In the pre questionnaire the students became aware of what was expected of 
them on the conceptual level. This type of learning activity was of a reflective nature. This awareness could 
also help the students to focus on important concepts during the semester. 

The other type of learning activity we promoted in the course was active participation while constructing 
interactive systems. The knowledge achieved by the students was expressed in actual designs of prototypes 
and reflections on these. Part of the knowledge expressed in action and in the design of prototypes will often 
be difficult to put into words and can be described as tacit knowledge (Schon, 2001; Agyris, 1978). 
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The concept of knowledge-in-action alone is not sufficient in a learning process or in a field practice. This 
must be supported by the more retrospective forms of reflection - reflection-on-action. Reflection-on-action 
helps the students to articulate conceptual knowledge on app-programming and HCI. In the retrospective 
reflection process their own experiences are connected to emerging conceptual knowledge. And conceptual 
knowledge is used in the professional communication amongst peers. 

In the classroom we wanted both knowledge-in-action and reflection-on-action. It is in the interplay 
between these forms of reflections that the skilled designer unfolds his potential. It is in this interplay that 
innovative processes evolve. It is also in this interplay that the students achieve a good learning depth. 

To put extra focus on the reflection-on-action the students made a self-assessment at the end of the course 
similar to the one at the beginning. In this assessment the students again were to reflect on their 
understanding of core concepts in HCI. This type of reflection was not so much linked to optimising their 
interactive designs, it was more linked to their understanding of the academic concepts in HCI. This type of 
reflection is important and productive in higher education. The students must understand the academic 
concepts in depth in order to develop a critical and analytical approach to them. 

The testing tool - the questionnaire - used in this work was based on an idea of one of the authors. The 
primary goal was to make it measure the learning using the SOLO Taxonomy (Biggs and Tang, 2007), which 
divides learning into a hierarchy of 5 levels: prestructural (misses point), unistructural (identify, do simple 
procedure), multistructural (enumerate, describe, list, combine, do algorithms), relational (compare/contrast, 
explain causes, analyse, relate, apply) and extended abstract (theorise, generalise, hypothesise, reflect). The 
SOLO Taxonomy is often used to form intended learning outcomes for any given course using this hierarchy 
of verbs, but here we use it to form the grading system presented in the methods section. 

Our main objectives for this study were to design and evaluate a simple self-assessment tool for pre- and 
post-testing in a first- semester engineering course. 


2. METHODS 

The research methodology was based on Design-based Research and action research (Barab and Squire, 
2004; Lewin, 1946; Majgaard, 2010; Majgaard et al, 2011). Design-based Research is a branch of 
educational research that uses the iterative design of educational interventions to exemplify and develop 
theories of learning. It also brings a change in the behaviour of the target group into focus and allows 
emerging goals. Experiments and critical reflections are the core of the research method, allowing learning 
from and through practice. In this research the experiments covered self-assessment before and after the 
course. 

The target group of this study is first-semester Software and IT engineering students. The main academic 
learning objectives of the 5-ECTS Interaction and Interaction Design course are that the student at the end of 
the course should be able to: 

• Plan a user-centred design process 

• Investigate the users, their needs and their practice through interviews and observations, and present 
the results in ways suitable for making design decisions. 

• Involve users in design and evaluation in suitable ways 

• Design interactions to fit with the users’ needs and practices 

• Apply fundamental design rules for user-friendly designs 

• Describe different types of interactions 

• Use selected types of interactions 

• Plan and conduct evaluation of interaction design and present the results 

• Develop simple digital prototypes 

• Reflect on interactive design processes and the meaning of good design 

The book used in the course is Interaction Design: Beyond Human - Computer Interaction. (Rogers et al, 
2011). The course was followed by approximately 70 engineering students and run in parallel with their 10- 
ECTS semester project on user-centred design allowing them to work more thoroughly with the theories and 
methods of the course. Besides the pre- and post-tests the students had an oral examination. 
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The questionnaire, which is used for both the pre and post student self-assessment in this course is based 
on a likert-type (Rensis, 1932) SOLO-inspired (Biggs and Tang, 2007) grading of the students’ assessments 
of their own learning. The questionnaire is comprised of 50 relevant terms within the subject areas: 
interaction design and user-centred design. The terms are chosen to broadly cover the learning objectives of 
the course as well as the content of the course book and can be seen from Figure 1 . We are not going for a 
specific number of questions, but rather want to make sure we cover the essential parts of the course material. 
With our questionnaire, we have three aims: the first is to get an overall measure of the students’ prerequisite 
knowledge of the selected terms. The other is to get an indication of the learning taking place during the 
course through a comparison of the results of the pre and post questionnaires. Our third aim is to get the 
students think about their own knowledge of the different subjects in this course through self-assessment 
reflections. 

The succession of the terms of the questionnaire are closely related to the succession of the subjects of the 
book and the course plan, and as such lower-numbered terms are presented early in the course and high- 
numbered terms at the end of the course. 

The questionnaire is handed to the students in paper form at the beginning of the first of a total of 12 
lessons within the course. The students are asked to fill in the form grading each of the 50 terms using the 
following numbers: 

1 . 1 have never heard of this before. 

2. 1 have heard of this before, but do not really understand what it means. 

3. 1 have an idea about what this means, but 1 don’t want to have to explain it. 

4. 1 have a clear idea about what this means and 1 am able to explain it. 

5. 1 know exactly what this means and 1 am also able to relate it to other subjects. 

The data of the filled-in and anonymous questionnaires are collected and entered into a spreadsheet, 
where the average grading of each of the subjects is computed and visualised in a bar graph. The average 
grading results are analysed and from these eventual changes can be made to the course curriculum. 

The post questionnaire is handed out at the last lesson of the course before the examination preparation 
period and the procedure is exactly the same as for the pre questionnaire. The average grading of each of the 
terms of the pre and post questionnaires can now be compared and we use this as an indication of the 
teaching and learning results of the course. 


3. RESULTS 

With the pre- and post questionnaires, we attempted to measure the student’s self-assessed learning at the 
beginning and at the end of the course. 

3.1 The Pre Questionnaire 

62 students answered the pre questionnaire, and looking at the results of the average grading of the terms in 
this test (the dark grey bars of Figure 1), we see that the responses to the 50 terms are widely spread from just 
above 1 up to ~4.6. The average is 2.47, stating that if we look at all the student answers, the students would 
rate their own knowledge as “1 have heard of this before, but do not really understand what it means”. 

3.2 The Post Questionnaire 

43 students answered the post questionnaire, and now the average grading of the terms (the light grey bars of 
Figure 1) range from just below 2.50 up to 4.60. The average is 3.82, which means that overall the students 
would rate their understanding of the course as “1 have a clear idea about what this means and 1 am able to 
explain it”. 

3.3 Comparison 

The question terms that show the smallest gap between pre and post are e.g. “Questionnaires”, 
“Brainstorming” and “Evaluation”. The question terms that show the largest gap between pre and post are 
e.g. “Wizard of OZ - model”, “High Fidelity / Low Fidelity” and “Life -cycle model of Interaction Design”. 
The average rise from pre to post is 1 .36. 
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Figure 1. Results of Pre and Post Questionnaires in the Course Interaction and Interaction Design, 2012 
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4. DISCUSSION 

4.1 Analysis of the Results of the Pre Questionnaire 

As anticipated from the questionnaire design, the students rate the terms that are commonly known to people 
at a higher level of understanding. “Questionnaires”, “Observation” and “Brainstorming” are rated above 4 
followed by “Good Design / Bad Design”, “GUI”, “Focus Groups”, “User Involvement”, “Prototyping” and 
“Evaluation” which are all between grading 3 and 4. We were surprised to see “Focus Groups” and “User 
Involvement” rated at this level, since we did not expect first-semester students coming from pre-university 
educations to have any knowledge about this. 

At the lower end of the self-assessed level of understanding, we see subjects such as “Wizard of Oz”, 
“Anthropomorphism“, and “Distributed Cognition”. These are clearly very specific subject terms and looking 
through the data there are only a few students that rate these higher than level 3. Overall the students have 
never heard of these terms before. 

The result of the pre questionnaire only led to a few changes in the curriculum of the course. First of all, 
there were only few surprises going through the statistics, and secondly, the spreading of the results of the 
individual questionnaires was for most subjects large enough that it would still be necessary to let the 
students work with a given subject in order to be able to reach the same level of understanding. What we did 
change based on the pre-test was, however the focus on brainstorming within the projects. Here, a large 
majority of the students rated this high enough to make us assured that they would be able to do proper 
brainstorming in their project groups. And since the project groups consisted of 5-6 students, we believed 
that those who did not have any experience in brainstorming would gain this from their fellow students 
within the groups. If the students scored four in brainstorming, they agreed on being able to explain the 
concept. This isn’t the same as being able to implement a brainstorm and it isn’t the same as knowing 
specific brainstorming methods. But from a debate in the classroom we came to the conclusion that 
brainstorming was something that most students had actually worked with beforehand and the project groups 
were all quite confident on how to proceed with this. 

From the lecturer’s perspective the pre questionnaire resulted in minor adaptations of the course. This can 
be compared to Schons reflection-in-action where the lecturer adapts on the fly his curriculum (Schon, 2001; 
Argyris 1978). 

From the students’ perspective the pre questionnaire gave them a possibility to reflect on the important 
academic concepts in the HCI-course. The students also obtained a small insight into the gap of what they 
already knew and the knowledge they were supposed to be familiar with at the end of the course. They might 
also have obtained some of the same knowledge while browsing in the semester book on HCI. In addition, 
the questionnaire gave them time to reflect on their own academic knowledge level and the expected 
knowledge level. This can be described as specific knowledge gaps (Angelo and Cross, 1993). 

4.2 Analysis of the Results of the Post Questionnaire and Comparison from the 
Lecturer’s Perspective 

The calculated averages of the post questionnaires show increased levels of self-assessed understanding for 
all of the terms except one. Luckily, no negative learning was observed at any of the terms. This was first and 
foremost of great reassurance in terms of the lecturer of the course, but it also gave a good overview of where 
the students had learned the most and the least and how this would potentially change the curriculum of the 
course this year and the following. From the results obtained, it is quite obvious that the students have been 
honest about their learning. Preferably, the lecturers would have liked to see an average for most of the terms 
just around or above 4, stating that the students would be confident enough with each term in order to be able 
to explain it and maybe relate it to other subjects inside or outside the curriculum. In the following we have 
picked out a few of the most interesting terms from Figure 1 and analysed them further. 
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4.2.1 “1. Interaction Design” 

This term represents the overall subject of the course, and as such it is important that the students leave the 
course with a high degree of knowledge about what this term means and also an ability to explain it to others, 
and in the case of their semester project, an ability to to work with the different parts involved in doing 
interaction design. The students enter the course with a knowledge grading of this at an average around 2.70, 
which means that they probably read the study plan before going to the first lesson, so that they have an idea 
about what interaction design means and what the contents of the course are going to be. At the last lesson 
this grading is very close to 4.50, which is considered very good indeed and which assures the lecturers that 
at least an overall understanding of the course contents has been achieved. 

4.2.2 “4. User Experience and User Experience Goals” 

Setting goals for and assuring specific user experiences are some of the most important aspects of doing 
interaction design today, so a good understanding of these terms is a must and a necessary prerequisite if you 
want to work at a professional level with interaction design. The pre questionnaire gives an average grading 
of these terms around 2.4 and the post questionnaire 4.00. Both usability and user experience have been 
central subjects in the teaching of the course and it is thus reassuring that the students rate themselves at a 
high level of understanding at the last lesson of the course. 

4.2.3 “6. Interaction Design Process” 

Looking at the learning objectives of the course the first objective is for a student at the end of the course to 
be able to “Plan a user-centred design process”, and the other objectives mention process keywords such as 
Design, Prototyping and Evaluation. In the course we have spent a lot of time teaching the four key elements 
of the interaction design process (Establishing requirement. Designing alternatives. Prototyping and 
Evaluation) and outlining that doing interaction design also naturally means doing user-centred design. It 
therefore comes as a surprise that the students do not reach an average grading of at least 4 for this subject at 
the end of the course, and therefore we will increase focus on this subject next semester. One analytical, but 
unsupported comment to the grading of this subject may be, that as the post test was run at the last lesson of 
the course - before the students’ preparations for the exam, and since the course material about the 
interaction design process is placed at the beginning of the course the students had less memories about this 
term and had troubles relating it to the actual content of it. 

4.2.4 “20. Questionnaires” 

As stated above, it was surprising for us to see “Questionnaires” rated at this level (4.56) at the pre 
questionnaire, and it did not leave much space for improvement in the post questionnaire (4.57). This clearly 
indicates one weakness with this type of measurement. When the students indicate a high knowledge level at 
the beginning of the course it is difficult to measure all the new material that they have worked with during 
the course (e.g. types of questionnaires, different ways of asking questions, measuring quantitatively or 
qualitatively etc.). 

4.2.5 “26. User Involvement” and “27. User-Centred Design” 

These two terms are central to the learning objectives and very much related to the overall interaction design 
process and within the teaching there has ben great focus on stating the importance of the user and the 
necessity of involving them in the design process in order to assure usability and the right user experience. It 
was nice to see these as rated above 4 with the post questionnaire. 

4.2.6 “42. Wizard of Oz - model” 

The Wizard of Oz term rates, as expected, as unknown to the students in the pre questionnaire. This does, 
however, change dramatically during the course, and it receives an average rating of 4 at the end being the 
one term for which the students’ average knowledge level changed the most. Wizard of Oz is a term for a 
model of how to simulate interaction with very simple and usually non-interactive prototypes, and thus the 
term cover a very important part of doing initial prototype evaluation at a stage of design where it is still too 
uncertain to start spending time developing technologies. This term is therefore regarded an essential part of 
interaction design and therefore it is very important that it is highly rated in this questionnaire. 


155 



ISBN: 978-989-8533-18-0 © 2013 lADIS 


4.2.7 “45. Participatory Design” 

Within the course and especially the students’ semester project we did not ask the students to involve the 
users directly in the design of the prototypes. Therefore, and as an undesired result, the students regarded this 
term as less important within the curriculum even though it actually is an important part of interaction design 
that the designer has knowledge about and is able to choose between different methods of user involvement. 
Learning of this term will as a consequence be reinforced within the next semester course. 

The above was just a small selection and examples of terms and subjects that have allowed us to evaluate 
this course through the pre and post questionnaires and the students’ self-assessment. The following is a list 
of numbers of the terms and subjects that will receive increased focus in the course next semester: 5-15, 29- 
31, 43 and 45. Hopefully this will be possible without influencing some of the other terms in a negative 
direction. From the lecturers’ perspective the post assessment was aimed at optimising the course for the next 
throughput. E.g. the lecturer wanted to increase focus of concepts: 5-15, 29-31 etc. This can be compared to 
Schon’s reflection-on-action which is the subsequent reflection and evaluation on the process that has 
happened, and its potential consequences (Schon, 2001; Argyris 1978). It is precisely this type of reflection 
you want before running this new course for the second time. 

4.3 Analysis of the Results of the Post Questionnaire and Comparison from the 
Students’ Perspective 

From the students’ perspective the post questionnaire gave them a possibility to reflect on their current 
academic level at the end of the course. They could also compare their current level with their initial level 
and reflect on their progress. Furthermore, the test provided the students with a chance to evaluate their own 
academic level before preparing for the final exam. These reflections are of adaptive nature and can be 
compared to Bateson’s 2"^ level of learning which provides for good and normal learning (Bateson, 2000). 
But the students also had the possibility to use the test results as a foundation for preparing their exam. In this 
way the results made the foundation for a possible layout of a learning strategy. A change in a student’s 
learning strategy can be regarded as level-3 learning. This type of learning provides a more enhanced form of 
learning. Students in higher education can benefit from reflections on own learning strategies (Qvortrup, 
2006; Gleerup, 2003). 

4.4 Evaluation of the Questionnaire 

As the above analysis of selected terms demonstrate it has been quite easy to reflect on the contents of the 
interaction design course using the calculated and graphically represented results (Figure 1) of the pre and 
post self-assessment questionnaires presented in this paper, and we can see no reason why this should not 
apply to other types of courses as well. Using the SOLO Taxonomy for grading knowledge levels of specific 
course terms through self-assessment is to the best of our knowledge a new approach on measuring learning 
and evaluating teaching. Others (reference) have used pre- and post-test student self-assessment for 
measuring teaching quality but with different test methods. 

4.4.1 Strengths 

What our investigations showed was that this method is easily implemented also in large classes where it 
becomes a lengthy and often tedious process to pre-test students using traditional assessment techniques. The 
production of the questionnaire with relevant course terms and subjects can be prepared at the same time as 
the course plan is made. An electronic version of this kind of questionnaire will be examined next semester, 
which will speed up the handling of the data. 

The process of comparing the results of the pre and post questionnaires becomes easy because with this 
method we are able to hand out the exact same questionnaires in both cases and the results can be directly 
compared and evaluated. With non self-assessment tests, such as e.g. multiple -choice having the same test 
two times during a course would potentially affect the students’ answers and as a consequence influence the 
validity of the test. 
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With regards to the students’ self-assessment there is theoretical background (references) for saying that it 
is going to be beneficial for their understanding of the course material that they are reflecting on their own 
learning of the most relevant course terms and subjects two times during the course. We do, however, have 
no measurement of the effects of this yet. 

4.4.2 Limitations 

The results showed in this paper prove an increase in the measured levels of understanding for the different 
terms stated in the questionnaire, which indicates the validity of our method, but further research needs to 
determine this. We do not yet know if the SOLO-inspired rating used in the questionnaire is optimal for this 
type of self-assessment. Would it e.g. be beneficial to have more or less levels? Would ratings inspired by 
Bloom’s taxonomy (Bloom and Krathwohl, 1956; Anderson and Krathwohl, 2001) result in more valid 
results? Also, retropectively the self-assessment should focus more on one’s ability to work with a given 
subject, because knowing exactly what a term means does not necessarily make you able to actually use it 
and implement that knowledge e.g. in developing new interactive technologies. In order to effectuate this, the 
wording of the grading system will have to be changed. 

Another limitation of our method is that we do not capture the increased knowledge that also takes place 
with the terms that are rated high (4-5) from the beginning. This has been mentioned previously and is 
something we will have to take into consideration when we formulate our questionnaire next semester. We 
may avoid using commonly known terms in the questionnaire, or at least put them in the context of course- 
relevant terms. 

A last limitation in the type of self-assessment that students do through our questionnaire is that it 
becomes difficult to measure any misconceptions the students may have regarding the different learning 
elements. Both at the beginning and at the end of the course. 

4.4.3 Comparison with Other Studies 

When comparing our study to the study mentioned in the introduction (Schiekirka, 2013), the two studies 
clearly differ in the grading system used. They use 6 levels of agreement to the understanding of a given 
learning objective, while we use SOLO-inspired 5-level grading system to measure understanding of a given 
course term. Both methods seem valid, but in our opinion the SOLO-inspired grading that we use is 
specifically formulated for each grading and make it easier for the students to pinpoint the exact grading and 
better reflect on their learning. 


5. CONCLUSIONS 

In this paper we explored how student self-assessment can be used as a tool and become beneficial for both 
lecturers and students. We used a simple self-assessment tool for pre- and post-testing in a first semester 
engineering course. The students graded their knowledge on human-computer interaction based on their 
ability to understand and explain specific concepts. 

Generally the assessment tool promoted practice reflection both reflection-in action and reflection-on- 
action. In the pre-test the students became aware of specific academic expectations in the course and they got 
to grade the gap between their own current knowledge and what would be expected at the end of the course. 
The lecturer could right from the beginning sharpen the academic semester plan based on the assessment 
results. 

The post-test could be used by the students as a tool for grading their knowledge in preparation for the 
final exam. From the lecturers’ perspective the post assessment was useful in optimising the course for the 
next throughput. 

The lecturer has decided to use the self-assessment tool again with minor adjustments. The grading 
system has to be adjusted into being more application oriented. The students should be able to grade their 
ability to use, apply, analyse, create or design according to specific concepts. This could easily be done 
within the scope of SOLO. 
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ABSTRACT 

Learning should make information meaningful to the students. We can all agree with that -hut the messages coming from 
schools will tell us a different story. Instead of enthusiasm, happiness and joy of learning, the echo repeats problems, ill 
behaving and indisposition. This makes us wonder: has the traditional school come to its end? 

This article examines pedagogical changes in different learning contexts with the help of the model of Contextual 
pedagogical approach towards learning (Merilainen & Piispanen 2012; Merilainen, Piispanen & Valli 2013). When 
working in learning environments based on Contextual pedagogical approach towards learning, the teacher will 
emphasize trans- disciplinary approach to curriculum, enhance student’s individuality and creativity in different learning 
situations and support both content knowledge as well as 2L‘ Century civil skills knowledge to develop hand in hand. 

The 21st Century Civil Skills Pedagogical Content Knowledge -framework, introduced in this article, will help teachers 
to create learning environments where 21st century civil skills will meet the modern pedagogy and core curriculum 
standards. With the help of the framework, together with the model of Contextual pedagogical approach towards 
learning, one is able to build a strong pedagogical foundation to create transformational 21st century learning 
environments. 


KEYWORDS 

Learning environment. Transformational pedagogic. Contextual pedagogical approach towards learning, 2L‘ Century 
CSPCK 


1. LEARNING FOR SCHOOL -LEARNING FOR LIFE 

In recent years, educators and policy makers have been focused on student achievement and well-being. 
After the great Pisa -success and glory, Finnish researchers have risen up discussion about future skills, 
school activity and motivation - fields where Finnish students haven’t indicated so well. The problem in 
Finnish school, according to Valijarvi (2011), is that despite the knowledge and skills Finnish students have, 
they donT trust to their know-how and their attitude towards learning is poor. It is worth asking, could you 
find any answers by looking deeper into our pedagogical choices, operation culture as well as learning 
environments in Finnish education system? 

Quickly changing 2F‘ century challenges teachers to see life outside the school and recognize not only 
the core subjects but also the key skills needed outside there. The report Learning for the 2F‘ Century 
identifies nine types of learning skills, which are divided into three different key areas as following in Table 
1. In different learning contexts, in society which develops fast, the school should stay along in this 
development and should help students to learn not only the contents which arise from the curriculum, but also 
the skills and matters that one needs in today's and future society, (cf. Levin 2011, 4; Zhao 2011, 4). More 
important than a huge amount of detailed information, should according to Merilainen & Piispanen (2012) be 
the multidimensional education, which comprises the know-how of different skills to make a good use of 
curriculum general information. If you look at students born in late 90s and early 2000, you can see a huge 
gap between the knowledge and skills students learn in school and the knowledge and skills they need in 
typical 2F century communities and working places. 
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Today's education system faces irrelevance unless we bridge the gap between how students live and how 
they learn. Moving from content knowledge to learning and life skills is essential when training students to 
be successful in their lives after school. 


Table 1. 2T‘ Century Learning Skills. 


1 21'‘ Century Learning Skills | 

INFORMATION AND 

THINKING AND PROBLEM SOLVING 

INTERPERSONAL AND SELF- 

COMMUNICATION SKILLS 

SKILLS 


DIRECTIONAL SKILLS 

Information and Media Literacy Skills 

Critical Thinking and Systems Thinking 

Interpersonal and Collaborative Skills 

• Accessing and managing 

• 

Exercising sound reasoning. 

• 

Demonstrating teamwork and 

information. 

• 

Making complex choices. 


working productively with others. 

• Integrating and creating 

• 

Understanding the interconnections 

• 

Demonstrating and the ability to 

information. 


among systems. 


adapt to varied roles and 

• Evaluating and analyzing 




responsibilities. 

information. 

Problem Identification, Formulation & Solution 

• 

Exercise empathy and respecting 

Communication Skills 

Ability to 



diverse perspectives. 


• 

frame 

Self-Direction 

Understanding, managing, and creating 

• 

analyze 



effective communications 

• 

solve problems. 

Monitoring one’s own understanding and learning 

• orally 


needs 


• written 

Creativity and Intellectual Curiosity 

• 

Locating resources 

• using multimedia 

Develop 


• 

Transferring learning from one 
domain to another. 


• 

implement 




• 

communicate 

Accountability and Adaptability 


• 

new ideas to others. 

• 

Exercising personal responsibility 
and flexibility in personal, workplace 
and community contexts. 




• 

Setting and meeting high standards 
and goals for one’s self and others. 




Social Responsibility 




• 

Acting responsibly with the interests 
of the larger community in mind. 




• 

Demonstrating ethical behavior in 
personal, workplace and community 





contexts. 


2. BROADING THE TEACHER S EXPERTISE 

The teacher's challenge in today’s education is to strengthen the students' natural ways to learn and produce 
information in new learning environments. Learning is thus seen as something happening in connection with 
an individual and his or her environment. Norrena, Kankaanranta and Nieminen (2011) argue that there has 
to be a significant pedagogical change in school routines and pedagogical operations to move from teaching 
to learning and towards 21’’“ century requirements. How will this change become true in school contexts - 
what are those pedagogical changes in the field of curriculum, planning and implementing as well as the roles 
of teachers and students? The Figure of 21’’“ Century Civil Skills Pedagogical Content Knowledge (21“'' 
Century CSPCK) (Fig. 1) attempts to identify the nature of vast pedagogical knowledge required when 
turning learning from traditional to transformational i.e. blending the 21st century civil skills in to the 
authentic learning contexts and the curriculum. 

The basis of the framework is the understanding that teaching itself is a highly complex activity that 
draws on many kinds of knowledge. This knowledge, as Ashe and Bibi (2011) highlights, is diverse and 
includes both content and pedagogical knowledge. In recent years the new type of knowledge has been raised 
to attention that of 2F‘ century skills or 2F‘ century civil skills as Finnish National Board of Education has 
named that knowledge in curriculum renewing process. The 2L‘ Century CSPCK -figure articulates the role 
of 2L‘ century civil skills in the process of teaching and learning in a really blended manner. In 2 1 "' Century 
CSPCK -model the emphasis is put on competency, performance and capabilities and the key question in 
learning situations is rather how the information will be used than what the information is. 
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21«OV1L SKIUS 
PEDAGOGICAL 



knowledge 

^ contexts ^ ^ • 


21st century ' 
ctvil skills 
content 
knowledge 


\ 


Eigure 1. The 21st Century Civil Skills Pedagogical Content Knowledge (21st Century CSPCK). (Eollowing 
Mishra & Koehler 2006, 2009) (Piispanen & Merilainen 2013). 

At the heart of the 2 1'*' Century Civil Skills Pedagogical Content Knowledge framework according to 
Piispanen & Merilainen (2013) is the complex interplay of three primary forms of knowledge; 2C‘ Century 
Civil Skills Knowledge (2C‘ Century CSK), Pedagogical Knowledge (PK), and Curriculum Content 
Knowledge (CCK). It is essential to find the 2C‘ Century Civil Skills Pedagogical Content Knowledge point 
of intersect, where the three primary forms of knowledge meets each other and use that essence as a starting 
point when creating innovative and enthusiastic learning situations, (cf. Mishra & Koehler 2006, 2009) As 
Merilainen & Piispanen (2012, 2013) highlights, the planning process is to be viewed from at least three 
different angles as pictured in Fig. 1 . What do we mean by that is that the emphasis of learning should not lie 
on curriculum contents (subject contents) themselves, but these contents should act as tools for 
accomplishing 2F‘ century civil skills by arranging learning situations and environments as authentic as 
possible to support vast and deep understanding of every day phenomena. The 2C' century civil skills should 
also not be seen as isolated skills or learning targets, but they should be examined as visible parts of a 
learning context. Together all the three knowledge areas will create a successful and pedagogically 
meaningful learning process produced by students and supported by teachers. 


3. FROM TRADITIONAL TO TRANSFORMATIONAL 

When transferring from traditional education to transformational, one has to imagine new ways to think about 
teaching and learning. According to Chaltain (201 1) traditional schools assume the student bears the primary 
responsibility for learning while transformational schools shares that via a learning team that includes, and 
extends beyond the teacher and student. In terms of student achievement, a traditional school emphasizes test 
results instead of students' aspirations and life options which transformational school focuses on. In 
transformational school, the target will be in working to build passion for learning in all students. 

Student achievement is a primary focus in all teaching and learning situations. Learning experiences 
should accord to Drake and Burns (2004) be relevant to student’s interests. When students are engaged in 
learning, as the writers highlight (2004), students will manage well in multiple academic areas. 

When moving from traditional pedagogy towards transformational education the use of 21st Century 
CSPCK -framework as a ground of learning will expand the learning process to include the 21st century civil 
skills knowledge as one of the three key elements in all planning, teaching and learning. 
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The difference between traditional pedagogy (subject or theme based learning) and transformational 
pedagogy (contextual pedagogical approach to learning) lies all the way from planning to implementing on 
the roles of the curriculum, teacher and student as well as learning tasks and learning environments as 
Merilainen & Piispanen (2012) states. 

Traditional, subject centered or multidisciplinary integration, which is commonly known as theme- 
teaching, focuses primary on the disciplines. The curriculum contents are integrated around a theme raised 
commonly from the curriculum. In this traditional pedagogical model one can recognize different disciplines 
and the working methods and the operation culture is often based on the use of text books, traditional 
learning tasks and concentrating on the content rather than skills. Learning outcomes are usually similar and 
learning situations are teacher centered. The assessment happens mainly in the end of the learning unit and 
the common way to collect relevant evaluation material is tests. The Table 2. collects together essential 
features of traditional teaching and learning from the teachers and students point of wives from the beginning 
of the planning process till the end of the process. 

Table 2. From planning to assessment in traditional pedagogical model (Piispanen & Merilainen 2013). 


From planning to assessment 

Teachers role 

Students role 

BASIS FOR PLANNING 

Core curriculum, text books, teacher 
handbooks 

Not involved 

TOOLS FOR PLANNING 

Different subject contents 

Learning materials (books, text books, learning 
games, etc.. 

Teacher handbooks 
Schedule 

External structures 
Multidisciplinary approach 

Not involved 

LEARNING SITUATIONS 



in the beginning of the process 

Teacher driven, group instructions 

Teacher has the knowledge -knows what is 
meant to learn and how 

Teacher presents the learning case 

Teacher centered 

Receiving information 

Acting according to teacher driven instructions 
Using material given by the teacher 
Working with text books and materials made 
by someone else 

Everyone working according the instructions 
approximately at the same time 

Talented students will have additional tasks 

during the process 

The interaction in the classroom from teacher 
to student, from student the teacher 

Weak students will do less or leave tasks 
unfinished 

in the end of the process 

Testing the knowledge with self-made or ready 
make tests. Gives feedback with test numbers. 

Everyone has done well-nigh the same tasks in 
the end of the process 

ASSESSMENT 

Teacher knows the assessment criteria 
Teacher provides feedback 

The final assessment is based on activity, 
outcomes and test 

Teacher emphasis the assessment of learning 

Students don't know the assessment criteria 
Students will get the information about 
learning by doing tests 

Students are divided to weak and good learners 
according to success in different tests 

From planning to assessment 

Teachers role 

Students role 

BASIS FOR PLANNING 

Core curriculum, text books, teacher 
handbooks 

Not involved 

TOOLS FOR PLANNING 

Different subject contents 

Learning materials (books, text books, learning 

games, etc.) 

Teacher handbooks 
Schedule 

External structures 
Multidisciplinary approach 

Not involved 
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LEARNING SITUATIONS 
in the beginning of the process 


during the process 


Teacher driven, group instructions 
Teacher has the knowledge -knows what is 
meant to learn and how 
Teacher presents the learning case 
Teacher centered 


The interaction in the classroom from teacher 
to student, from student the teacher 


Receiving information 

Acting according to teacher driven instructions 
Using material given by the teacher 

Working with text books and materials made 
by someone else 

Everyone working according the instructions 
approximately at the same time 
Talented students will have additional tasks 


Weak students will do less or leave tasks 
unfinished 


in the end of the process 


Testing the knowledge with self-made or ready 
make tests. Gives feedback with test numbers. 


Everyone has done well-nigh the same tasks in 
the end of the process 


ASSESSMENT 


Teacher knows the assessment criteria 
Teacher provides feedback 

The final assessment is based on activity, 
outcomes and test 

Teacher emphasis the assessment of learning 


Students don't know the assessment criteria 
Students will get the information about 
learning by doing tests 

Students are divided to weak and good learners 
according to success in different tests 


Highly structured and disciplined schooling systems do not necessarily prepare students well for the 
challenges of the future. The transformational pedagogy, as Merilainen and Piispanen (2012, Piispanen & 
Merilainen 2013) highlights, will contribute significantly to the preparation of a future workforce. 

Transformational, trans- disciplinary integration, which Merilainen and Piispanen (2012) calls Contextual 
Pedagogical approach to learning (Fig. 2), focuses on the three different knowledge areas as presented in 
Figure 1 . In the trans- disciplinary approach to integration, according to Drake & Burns (2004), a teacher will 
organize curriculum around the student questions and as Merilainen and Piispanen (2012, Piispanen & 
Merilainen 2013) adds, around the real life phenomena and operation cultures that rises from there. Instead of 
one discipline the examination is directed to the phenomenon at a trans- disciplinary point of view. With the 
dialog between the curriculum and surrounding real life, one will look for answers by thinking, by 
concluding and by examining, which will support the development of 2F‘ century civil skills simultaneously 
with the content knowledge. 

The contextual pedagogical approach, based on real life phenomena, is a way to examine the curriculum 
in the relation with the surrounding society. The curriculum and different subject contents will be examined 
transdisciplinary and one can understand the connections between the curriculum and surrounding society. 
As a result you can recognize and see the operation culture of the school reflecting the operation culture of 
the external world. (Merilainen, Piispanen & Valli 2013.) The curriculum will come alive as authentic as 
possible with real life tasks, roles and environments as mentioned earlier. In transformational model of 
pedagogy, students will naturally develop life skills as a norm. The learning tasks are similar to real life 
tasks. For example one will learn by planning guided tours around the city with the city guides. The emphasis 
is rather on the skills than on the content -both skills knowledge and content knowledge are to be learned and 
assessed. In the model of transformational learning the content will act as a tool for learning 2T'’' century civil 
skills. The assessment for knowing and understanding in transformational pedagogy is performance -based. 
Instead of testing the memory and seeking for one right answer, the assessment focuses on interdisciplinary 
concepts and skills and the culminating activity will reflect this. The assessment criteria are presented to 
students in the beginning of the project so that each student can and will do well on it. 

The Table 3. presents the typical features of transformational teaching and learning from the teachers and 
students point of wives from the beginning of the planning process till the end of the process. 
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Table 3. From planning to assessment in transformational pedgogical model (Piispanen & Merilainen 2013). 


From planning to assessment 

Teachers role 

Students role 

BASIS FOR PLANNING 

Students interests, surrounding society , 
real life habits and skills 

Strengths, students profiles 

TOOLS FOR PLANNING 

21st Century CKPCK 

Transdiciplinary approach 

Strengths, students profiles 


Real life experts 


LEARNING SITUATIONS 



In the beginning of the process 

Teacher as a motivator 

Personalized learning plans 


Presenting the project, mission, aims and 

Students themselves set the goals 

During the process 

assessment criteria. 

Leader of the learning community 

Feedback by discussing with learners 

Supporting when needed 

Aware of each child's strengths and 

weaknesses, willing and able to support 

during the learning process 

Support towards the goals 

Students working collaboratively as a team 
Multiple ways to show learning 
Multiple use of ICT 

Learning by working with real life task in real life roles with real 
life experts 



Students build exhibitions, festivals, workshops etc. to show 


Discussion of learning 

Assessment as learning 

what they have done during the learning process 

In the end of the process 



ASSESSMENT 

The assessment criteria will be visible 

Students are aware of the assessment criteria from the beginning 


and presented to students in the 

of the process 


beginning with the process 

Students will get the information about learning discussing with 


Assessment as learning -towards life 

the teacher and peer students 


options 

Students are aware that they learn differently and that every child 


The assessment is an integral part of the 
learning process 

can learn 

From planning to assessment 

Teachers role 

Students role 

BASIS FOR PLANNING 

Students interests, surrounding society , 
real life habits and skills 

Strengths, students profiles 


21st Century CKPCK 


TOOLS FOR PLANNING 

Transdiciplinary approach 

Strengths, students profiles 


Real life experts 


LEARNING SITUATIONS 



In the beginning of the process 

Teacher as a motivator 

Personalized learning plans 


Presenting the project, mission, aims and 

Students themselves set the goals 

During the process 

assessment criteria. 

Leader of the learning community 

Feedback by discussing with learners 

Supporting when needed 

Aware of each child's strengths and 

weaknesses, willing and able to support 

during the learning process 

Support towards the goals 

Students working collaboratively as a team 
Multiple ways to show learning 
Multiple use of ICT 

Learning by working with real life task in real life roles with real 
life experts 



Students build exhibitions, festivals, workshops etc. to show 

In the end of the process 

Discussion of learning 

Assessment as learning 

what they have done during the learning process 

ASSESSMENT 

The assessment criteria will be visible 

Students are aware of the assessment criteria from the beginning 


and presented to students in the 

of the process 


beginning with the process 

Students will get the information about learning discussing with 


Assessment as learning -towards life 

the teacher and peer students 


options 

Students are aware that they learn differently and that every child 


The assessment is an integral part of the 
learning process 

can learn 
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3.1 In Relationship with the Authentic Contexts 

Piispanen & Merilainen (2013) argues that in a Contextual-pedagogical approach towards learning, a special 
attention is paid to the growht of 21st Century CSPCK knowledge. The skills, context and pedagogic will 
have a crucial significance in all learning situations. Where traditional pedagogic and multidisciplinary 
approach to integration emphasizes pedagogy and curriculum as tools for creating learning situations, the 
transformational pedagogic connects the three knowledge areas together. The learning situations will be 
discovered in the heart of the expanded knowledge acquisition as you can see in Fig. 2. 

CONTEXTUAL - PEDAGOGICAL APPROACH 
& TO LEARNING 



\ 


Eigure 2. Contextual - pedagogical approach to learning (Merilainen & Piispanen, 2012.) 

The pedagogical planning focuses on the child and his uniqueness which influences to the choices that a 
teacher will do concerning the learning context as well as the pedagogics, contrary to traditional pedagogics 
where the choices are often driven by the school structures, schedules, classrooms and books. 

The basic idea behind the contextual pedagogical learning environment according to Piispanen and 
Merilainen (2013) lies on problem based; investigate learning in authentic learning environments with 
authentic learning tasks. The tearing can thus be seen learner centred, flexible, illustrative and suitable for 
many kinds of learners (See Piispanen 2008; Sahlberg 2011; Zhao 2011). All the suitable, appropriate to 
learning, places can be seen as good learning environments. It is possible also for students to create their own 
environment during the learning process. 

It is essential, in contextual pedagogical approach, to examine the curriculum transdisciplinary. In 
different learning processes, the boarders between different subjects will disappear and the focus will be in 
everyday phenomena that children are familiar with. It is the teachers responsible to see how the core 
curriculum is manifested in children's lives and to plan and organize learning situations motivated enough to 
make children devoted to given learning tasks. The teachers job is to act as a curriculum expert who is able to 
see the connections between the core curriculum and real life, a pedagogical specialists who is able to create 
enthusiastic learning situations, tasks and environments and finally the one who has a good student 
knowledge and who is willing and able to plan learning tasks and activities appropriate to each individual in 
one's class. In the very end it is the teacher who creates the learning tasks from pedagogically meaningful 
points of views and contacts the real life specialists who are able to give authentic learning experiences in 
authentic ways and act as co-operation partners in real learning situations. (See Kumpulainen & ah, 2011). 

It is essential to recognize, name and utilize learner’s versatile know-how in contextual pedagogical 
approach to learning. Every child should be able to use and show his best know-how and abilities as well as 
learn new skills from peer students and in teacher’s quidance while working in authentic learning situations. 
Student’s individual needs are seen crucial in contextual pedagogical learning process. Different learners will 
have possibilities to work at their best as well as show their best while working. 
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3.2 Bringing Enthusiasm to Learning 

In the Contextual-pedagogical approach towards learning, the essential change comparing with the traditional 
pedagogy concerns the students’ role as knowledge constructors: the culture of working largely alone with 
individual learning tasks is transferred to a culture of collaboration, high levels of collegiality, team work, 
and dialogue as a way of action. (Merilainen, Piispanen & Valli 2013.) 

Instead of just accomplishing the learning tasks, the students are directed to be active and self -piloting 
collaborative learners. This means a huge change in the traditional teacher -student -roles: the teacher will 
no more be the know-it-all -person, instead her/his role is to help students to address with information, to 
operate among the information and before anything, lighting the learning enthusiasm among the students. In 
this model, the teacher will see student's best potential and take risks to make that visible. 

It is essential to activate the students to work together so that the given tasks will support the 21*' century 
civil skills to develop. (Kostiainen & Rautiainen, 2011, 190). As Merilainen, Piispanen and Valli (2013) 
highlights, the learning tasks should be closely connected to student’s real lives, interesting, challenging and 
enable student's natural creativity and know-how to develop. It is outstandingly important that students have 
a possibility to act in roles, natural to learning tasks. That will motivate and help them to accomplish the 
tasks in the expected manner, similar to that in the authentic context. 

In the contextual - pedagogical model of learning, the 21*' century civil skills are not necessary the key 
objects of teaching, but their presence and use in different learning tasks will lay a solid foundation to deeper 
understanding, learning, knowing and creativity. (Hargreaves 2007, 223-224; Kumpulainen, Krokfors, 
Lipponen, Tissari, Hilppo, Rajala.2011, 46; Sahlberg 2011, 4; Zhao 2011, 2-3). When planning a learning 
process and paying attention to the development of these skills with other two knowledge acquisition areas 
(CCK and PK) will make it possible to create learning environments and learning situations that will support 
the 21*' century civil skills content knowledge to develop in a school context. 

3.3 Personal Learning Paths 

It is a central matter to pay attention to the students' individual needs in a Contextual- pedagogical approach 
towards learning. Transformational learning process enables diverse students to learn according to one's own 
best ability and to bring one's individual know-how visible. The paths toward set learning goals will be as 
unique as your students -the beforehand given goals and assessment criteria will guide students step by step 
towards the set goals. The paths will naturally become differentiated, never the less the learning has become 
true. 

When the curriculum contents are learned and experienced transdisciplinary in authentic learning 
conditions, students have possibilities to consider the given tasks multiple and visualize these in versatile 
ways. This gives an opportunity to emphasize individual learning styles and unique temperaments which are 
mostly seen as problems in our school system. 

The versatile examination of phenomena and the multiple choices of individual learning paths will create 
the possibility to learn and understand phenomena from student's individual premise in collaboration with 
others. 


4. LIVE-ROLEPLAYING THROUGH THE CURRICULUM 

What does this all, described above, mean in practice from the teacher's point of view? Let's look at the 
planning process closer. Where to begin, how to put emphasis on needed skills, what is the connection 
between subject contents and real life in practice, what means authentic learning environment? These are 
some of the questions that you as a teacher will have to pay attention to when moving from traditional 
pedagogic towards transformational pedagogical settings. Flipping the sight from society to curriculum puts 
emphasis of teaching curriculum contents to emphasis of learning real life phenomena and skills that we need 
in authentic learning environments and learning situations. The planning begins from the premise of 
individual student and his/ her skills, knowledge, interests and enthusiasm (Comparison to traditional 
planning where the planning is made to fit to the school constructions; timing, text books, classrooms, etc.) as 
you can see in Tables 2. and 3. Blending 21*' Century civil skills to the study plan becomes natural when the 
school tasks begin to remain real life tasks as you can see in Table 4. 
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Table 4. Contextual-pedagogical study plan in a nutshell (5th grade) (Piispanen & Merilainen 2013). 


Phenomenon (authentic/ outside the 
curriculum/ learning environment) 

Students role (authentic -rises from the 
phenomenon) 

Task (authentic -supports 2P* Century civil Skills to 
develop) 

To plan a Summer Camp in a Ranch 

Ranch owner / Camp director (live- 
roleplaying) 

To create an enthusiastic camp program, marketing plan, 
web & mobile pages and radio/ television commercial. 


In this model, as Merilainen and Piispanen (2012) states, the teacher holds the curriculum contents up to 
the surrounding world and connects the curriculum contents to real life phenomena. This will help students to 
understand and link the curriculum contents with the life outside of the school. The curriculum contents act 
as tools for developing 21^‘ century civil skills as explained in Figure 1. The Century CSPCK-framework 
will focus on a variety of different knowledge areas to develop both skills and content understanding. The 
pedagogical knowledge has to meet the century skills as well as the curriculum contents to be able to 
create learning situations, task and environments that will develop 21‘‘* century civil skills pedagogical 
content knowledge in a school context. 

Table 5. presents an example of a learning task, which will fit into the 21 century CSPCK-framework 
and illustrates the Contextual pedagogical approach towards learning concretely. The task is planned for 5^*^ 
grade students and the curriculum contents meet the 5^^ grade standards (Finnish National Core Curriculum 
for Basic Education 2004). 


Table 5. 5th grade curriculum contents related to given phenomenon and tasks. (Piispanen & Merilainen 2013). 


PHENOMENON: To plan a Summer Camp on the Ranch 

TASK: To create an enthusiastic camp program, marketing plan, web & mobile pages and radio/ television commercial 


CROSS CURRICULAR THEMES Media skills and communication. Participatory Citizenship and Entrepreneurship and 
Technology and the individual 


Mother tongue and 
literature: 
INTERACTION 
SKILLS 

The pupil will learn 
skills of active listening 
and communication in 
various communication 
situations; they will feel 
encouraged to take part 
in discussions and will 
try to consider the 
recipients in their own 
communication. 

The pupil will learn to 
work with text 
environments in which 
words, illustrations, and 
sounds interact 


Biology and 

Geography 

The pupil will learn 
to move about in the 
natural environment 
and observe and 
investigate nature 
outdoors 

The pupil will learn 
to draw and interpret 
maps, and use 

statistics, diagrams, 
pictures, and 

electronic messages 
as source of 

geographic 
information 


Music 

The pupil will build 
his/her creative 
relationship with 
music and its 
expressive 

possibilities, by 
means of composing 


Arts 

The pupil will learn to 
evaluate their own and 
other's visual 

expression and 

working approaches, 
such as visual, content, 
and technical 

solutions, and to 
employ the key 
concepts of art. 

The pupil will work 
independently and as a 
community member in 
art projects 


Mathematics 

The pupil will learn to 
understand that concepts form 
structures 


SKILLS IN 

PRODUCING TEXT 


The pupil will learn to 
create a variety of texts, 
both orally and in 
writing 

RELATIONSHIP 
WITH LANGUAGE, 
LITERATURE, AND 
OTHER CULTURE 


The pupil will gain a 
basic knowledge of the 
media and utilize 
communications media 
purposefully. 
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4.1 Assessment 

According to Finnish National Core Curriculum for Basic Education (2004, 260): The task of assessment 
during the course of studies are to guide and encourage studying and to depict how well the student has met 
the objectives established for growth and learning. It is the task of assessment to help the student form a 
realistic image of his/her learning and development, and thus to support the student 's personality growth, 
too. 

In transformational pedagogical settings the assessment will be seen as learning itself contrary to 
traditional pedagogic where assessment is seen as information of learning. According to Piispanen & 
Merilainen (2013) in the model of Contextual -pedagogical approach towards learning the assessing criteria 
will be visible and well known already in the beginning of the learning process. It is important to begin the 
planning process by identifying the learning goals and criteria for successful learning. After that the task and 
activity designing process is easy to accomplish so that the tasks and activities will help students to achieve 
these goals. Assessing will act as a tool for guiding students through the learning path -the learning aims will 
come true through the learning tasks based on assessing criteria. This is, as Merilainen & Piispanen (2012) 
states, particularly important in order to make students understand and recognize what are the learning 
expectations and how will the assessment come true. In essence, assessment occurs at all stages in the 
learning process and is beneficial for both the students and the teacher. 
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ABSTRACT 

This paper introduces four principles for the establishment of a technology-enhanced learning environment with and for 
children with autism spectrum disorders and presents results on how the principles were actualized in relation to 
children’s actions in the environment. The study was conducted as action research premised on the children’s active roles 
as participants and developers, the empowerment of children’s strengths and creativity, and the modifiability and 
transformability of technology solutions. The learning environment consisted of four workstations: symbol matching, 
building with bricks, storytelling, and game playing. According to results, the strength-based approach and versatile 
technology solutions engaged children with autistic features as active participants and creative actors. This engagement is 
crucial for creating new possibilities for the children’s education and everyday life. 


KEYWORDS 

Autism spectrum disorders, children, technology-enhanced, learning environment 


1. INTRODUCTION 

The contents of this paper reflect an ongoing research project that investigates the actions of children with 
autism spectrum disorders (ASD) in a strength-based technology-enhanced learning environment 
(Voutilainen et al., 201 1). The structure of the paper is twofold. First, the paper will introduce four principles 
for the establishment of a strength-based technology-enhanced learning environment, and second, it will 
present and discuss findings on how such a learning environment worked for children. 

The technology-enhanced learning environment included many technology solutions for children’s 
learning. Versatility of the technology solutions meant possibilities to foster children’s creativity and 
potential skills that a single technology solution might not have been able to emerge. With respect to its 
strength-based learning environment focus, this paper stresses the importance of establishing and developing 
a learning environment based on the strengths (e.g., special skills, interests) and creativity of children with 
ASD rather than on the problems and deficits associated with autism. This emphasis on strengths and 
creativity is important as these aspects have been less researched and understood than other features of 
autism (Happe and Frith, 2009). 

A learning environment’s characteristics, for example, class arrangements, computers, laboratory 
experiment kits, teaching methods, learning styles, and assessment methods, influence learners’ academic 
achievements and other learning outcomes in cognitive and affective domains (Doppelt, 2004, 2006; Doppelt 
and Schunn, 2008). The impact is even more remarkable when learners have special needs such as autism 
(Sze, 2009; Verdonschot et al., 2009; Williams, 2008; Williams et al., 2006). A growing number of studies 
suggest that interactive causal multisensory environments are stimulating for people with disabilities 
(Williams, 2008; Williams et al., 2006). In addition, recent research indicates that children with ASD, for 
example, benefit from environments that provide structure while allowing them to express their personalities 
in the learning choices they make (Sze, 2009). 
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There is evidence that an “autism friendly” environment needs to be based on individual assessment, 
focus on social understanding and communication, be developmental and structured, and use visual supports 
(Guldberg, 2010; Parsons et ah, 2009, 2011). Potential sensory processing difficulties need to be taken into 
account and environments adapted accordingly (Bogdashina, 2003; Frith, 2003). It is also important to 
consider a number of other dimensions, including teaching practices, learning contexts, and child 
characteristics, when building a supportive and activating learning environment for children with ASD. 


2. STRENGTH-BASED TECHNOLOGY-ENHANCED LEARNING 
ENVIRONMENT 

There were four main principles that established the learning environment in the research project: 1. 
Children’s creativity and active roles; 2. Children’s strengths; 3. Modifiability of technologies; and 4. 
Transformability of technological solutions to everyday life contexts. 

1. Children’s creativity and active roles as participants and developers in a technology-enhanced learning 
environment. This first principle investigated the diversity and creativity in children’s behavior - aspects that 
have been less researched compared to more typical features of ASD (e.g., repetitive and invariant behavior) 
(Napolitano et ah, 2010). The learning environment enabled the children’s active role by letting the children 
interact with many kinds of technologies. Technologies were selected to be diverse so that the children could 
use them in various ways through different kinds of interfaces (e.g., touchscreen, mouse, physical tiles, and 
motion-based interface). Various and changing pedagogical contents of technology applications (e.g., funny 
games, number and picture tasks, creating stories, building models) were to tempt the children’s engagement 
and creativity. 

2. Comprehensive support of the emergence of children’s strengths. The majority of research on children 
with autism and technology attempts to find solutions to problems connected to ASD (e.g., Austin et ah, 
2008; Bernard-Opitz et ah, 2001; Powers, 2006). This learning environment, however, focused on children’s 
strengths during activities in the environment. There were several ways to support the emergence of 
children’s strengths. The use of multimodal interaction, the utilization of different senses (visual, auditory, 
tactile, and kinesthetic), and the individual modifiability of technical solutions could help determine the 
children’s individual strengths. In addition, a roomy space with minimal external stimuli was to support 
children’s concentration on activities at workstations and give them a chance to monitor or to interact with 
other children while working in the environment. Also action group session routines (e.g. joint beginning of 
the session) were to enhance the clarity of the learning environment. However, as the environment was meant 
to be as natural as possible, changes in routines and the organization of the environment were possible when 
needed. 

3. Modifiability of technologies. This principle emphasized the children’s active role and creative actions 
in the learning environment. Pedagogical content and technological implementation of applications are often 
predefined before use in learning environments because they are often designed for specific purposes and 
certain learning objectives; therefore, children and teachers rarely have opportunities to modify physical 
technology devices or content. Technology solutions with specific purposes for children with ASD are for 
example mobile devices to improve communication skills (see De Leo and Leroy, 2008) and scheduling (see 
Hayes et ah, 2008), virtual learning environments and computer games for developing social skills (see 
Battocchi et ah, 2009; Cheng et ah, 2010) and games for exercising (see Finkelstein et ah, 2010), and 
robotics for improving social skills (see Fujimoto et ah, 2010). These technology solutions have indicated 
advantage for children with ASD within the specific purpose, but by enabling the modification of pedagogic 
content, the solutions might be applicable to other educational domains. 

The learning environment established in this research project realized the modifiability of technologies by 
enabling modification of physical elements (e.g. physical tiles) and pedagogical content (e.g. tasks and visual 
content) to applications by both children and adults. Choices for modification were based on the children’s 
interests and iterative feedback after participation at the workstations and observations of children’s action at 
the workstation. Thus, the participating children had an untraditional and unique role in the study since they 
operated as innovative and active research partners (Druin, 2002; Marti and Bannon 2009; Olkin, 2004) 
rather than just as objects of inquiry. The teachers’ and school assistants’ roles were also important in the 
development of the technologies since they knew the children’s individual pedagogical goals in school. 
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4. Transformability of technology solutions to everyday life contexts. Commercially available 
technologies (e.g., robotics) are often too expensive to use in education (Bryant et ah, 2010). Another 
obstacle to applying and transforming technology solutions to everyday life contexts is how time-consuming 
technologies are for teachers to learn and how difficult they are to use (Copley and Ziviani, 2004). Research 
on advanced technologies confirms children with ASD benefit from various technologies (Finkelstein et ah, 
2010; Williams et ah, 2002; Williams et ah, 2006) and thus supports applying technologies in education for 
them. It is therefore important that applications are easy to use and modify without technical expertise or 
external support to fit children’s needs and wishes in everyday life contexts like school. 


3. METHOD 

3.1 Participants 

Research participants included four children with autistic features in one comprehensive school for children 
with special needs. Two of the participating children were boys (ages 9 and 11) and two were girls (ages 8 
and 13). The children faced many challenges in their actions and learning, yet had multiple strengths, such as 
good visual senses, and a variety of ICT skills. Each child had limited verbal language skills but each had 
various ways of communicating. All of the children used augmentative and alternative communication 
methods, especially picture symbols, in various situations. 

3.2 Settings 

The research project ran one-hour group sessions, called an action group, weekly, nine times each semester. 
At the beginning of each session, there was a short warm-up with greetings and researchers gave the children 
a pictured map of the workstations. Though the order of the workstations was predetermined, the children 
could choose a variety of tasks or games to work with at each workstation. The children worked individually 
at each station for 10 to 15 minutes, and the adults were advised to help if needed (e.g., setting the difficulty 
level of the task). The order of the workstations varied for each child every session. 

A technology-enhanced learning environment was set up in a spacious room in the school building. There 
were four technology workstations in the learning environment: symbol matching, building with bricks, 
storytelling, and game playing (Figure 1). 



Figure 1. Symbol Matching, Building with Bricks, Storytelling, and Game Playing 


At the symbol matching workstation the children had tasks of matching a symbol from the computer 
application to the corresponding symbol or a theme by pressing on one of six tiles. The children chose the 
topic for the tasks and changed the symbol cards on the tiles according to their selection by themselves. At 
the building with bricks workstation the children built a LEGO® Duplo or basic LEGO® construction from 
the model on the computer application. The children chose a task from three alternatives: 1) building from 
the picture of the whole model; 2) step-by-step building of the model; or 3) a memory game that hid the 
model during the child’s construction. The children adjusted the difficulty level by changing the number of 
the bricks in the application. At the storytelling workstation the children created stories by using a picture- 
based computer application and a touchscreen. Pictures were categorized. Children created stories by 
dragging and dropping the hand-drawn pictures into the story’s timeline, and also by drawing pictures of 
their own. The stories were saved to the story library. The children could print out their stories and put 
together their own story books. At the game playing workstation the children played short Kinect 
Adventures! games by Microsoft Game Studios. Each child played the games by using his or her whole body 
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to control the game, for instance, jumping, dodging, and using their hands. Games were flexible, allowing a 
variety of movements as long as the player stayed within the play area. 

The children gave immediate feedback about the workstations after interacting with the technologies. The 
feedback system consisted of a black piece of cardboard with three picture-word feedback cards and a photo 
of the workstation. The feedback cards had drawn pictures of facial expressions (linked with matching 
words): very happy face (1 liked it a lot), neutral face (1 liked it a little), and sad face (1 didn’t like it). In this 
respect, the feedback scale was similar to one used with children in technology development projects using a 
participatory design model (see Nissinen et ah, 2012; Read et ah, 2002; Read and MacFarlane, 2006). 

3.3 Data Collection and Analysis 

The project conducted qualitative action research. The main research data were collected by videotaping each 
child’s actions using two video cameras per workstation to analyze the child’s actions while seeing what was 
happening on the screen at the same time. The additional data were collected by observing the children 
during the action group sessions. This paper’s results are based on data collected between February 2011 and 
May 2012. The researchers analyzed the data via content analysis by organizing and reviewing the data 
according to the four principles that guided the establishment of the learning environment. 


4. RESULTS 

According to results, the project’s strength-based technology-enhanced learning environment facilitated the 
emergence of the children’s activity and creativity. For instance, the children immediately started using the 
applications or choosing equipment linked to the workstations (e.g., cards for the tiles) upon arriving at the 
station and quickly learned compensatory ways to proceed if there were problems with technologies (e.g., 
using buttons on the keyboard instead of out-of-order tiles) or the equipment (e.g., using red bricks instead of 
missing orange bricks). All of the four children with autistic features showed strong interest in the new 
application features and new tasks or games in the environment, as the next example shows. 

At the beginning of the session, we presented a new task for the symbol-matching workstation called 
“Hypernyms task.” Iris, Ian, and Olivia chose the new task as the first task at the workstation. Eric 
scanned the new cards in his turn. The school assistant asked if he wanted to take the new task. Eric 
immediately started to place the new cards into the tiles. (Observation notes, March 2012, transcribed 
into English) 

The participating children’s charm of novelty was remarkable considering many researchers report that 
children with ASD have restricted interests (see Ala'i-Rosales and Zeug, 2008; Baron-Cohen and 
Wheelwright, 1999; Eolstein and Rosen-Sheidley, 2001). While the researchers executed changes in the 
learning environment based on routines familiar to the children, the children’s interest in change emerged 
from the beginning of the project, even when the workstations and procedures were novel to them. According 
to the findings, the versatility of the workstations in the environment and the possibility of making choices at 
each workstation seemed to support the active role of the children. In addition, similar to many previous 
studies (e.g., Einkelstein et ah, 2010; Williams et ah, 2002; Williams et ah, 2006), technology itself was 
extremely motivating for all participating children in the project. 

The role of the teacher or school assistant working with the child in the technology-enhanced learning 
environment was also significant in many respects. The teachers’ and assistants’ contributions were 
important in helping the children overcome possible problems in an application’s functionality or a task’s 
difficulty. In addition, the school assistant’s positive tutoring and feedback was relevant in helping the given 
child grasp a new task and learn to do the task by himself. 

Considering the results, the technology-enhanced learning environment brought out the children’s 
potential and strengths: the second principle in the establishment of the learning environment. As knowledge 
of the children’s strengths, and often of the children’s interests, was iteratively executed both in the content 
of the tasks and the workstations’ technical aspects, the environment kept changing and thus continuously 
fostered emergence of the children’s strengths. Their strengths varied from good visual perception to creating 
detailed drawings to athletic skills. Below is an example of one child’s skills in making choices 


173 



ISBN: 978-989-8533-18-0 © 2013 lADIS 


independently and moving fluently - skills which emerged especially in this environment since his actions 
were not very self-directed in the classroom setting. 

Ian trots to the play area at the game-playing workstation, chooses the first game, and plays it 
independently. Ian moves fluently and quickly to different directions during the game (stepping left 
and right, hands up, hands down, hands diagonally, stepping forward and backward, jumping) and 
collecfs lofs of poinfs. When he finishes the game, the school assistant says, “Really well, Ian, great,” 
and claps her hands. (Transcription of a video data clip, November 2011, translated into English) 

Overall, the teachers and school assistants working with the children in the action group reported 
remarkable differences, especially in the children’s attention spans in the classroom setting compared to their 
attention spans in the technology-enhanced learning environment, in favor of the last-mentioned. Even 
though this finding requires more thorough verification in the future, it is of great interest since researchers 
commonly report challenges in the attention spans of children with ASD (Bogdashina, 2003; Erith, 2003). 

The project iteratively realized and fulfilled the modifiability of technologies - the third principle of the 
establishment of the learning environment. The children’s iterative feedback played an important role in the 
modification process; however, the challenge at the beginning of the project was to get feedback from the 
children. Because the children were inexperienced in giving feedback, the researchers needed to carefully 
consider how to ask them for feedback. The feedback system described above seemed to work, as the next 
examples show. 

Olivia takes the feedback board by herself and the school assistant asks, “What did you like?” and at 
the same time Olivia says in a clear voice, “1 liked it a lot!” The school assistant confirms, “You liked 
it a lot.” Olivia then attaches the photograph of the workstation under the happy face and says again, 
“1 liked it a lot.” (Transcription of a video data clip, Eebruary 2012, translated into English) 

Iris immediately moves her finger straight back on the sad face. She points at it several times until the 
researcher names the picture, “1 didn’t like it.” (Transcription of a video data clip, April 2012, 
translated into English) 

Along with learning to give feedback for the workstations, the children’s participation in the modification 
process grew stronger. The children’s overall participation in the development of the project’s technology- 
enriched environment, however, was still limited and researchers need to develop more elaborate means of 
participation. The children’s feedback has nevertheless been a good starting point since it is unusual for 
children with ASD to be involved in the evaluation of their learning environments. 

The technologies’ modifiability relates to the fourth principle in the establishment of a learning 
environment: the transformability of technological solutions to everyday life contexfs. To be truly 
transformed to a school context, technologies have to be easy to use for both children and adults. If the 
application was too complex, the adult was not able to tutor the child on how to use the application 
appropriately or, therefore, to help the child perform the task purposefully. Clear instructions minimized the 
need for teachers and assistants to get support from technical experts or task designers and prevented 
misunderstandings of usage or technology content. It was helpful if the instructions were in sight in the tasks 
and games themselves and not hidden somewhere in the menus. According to the data, using only pictures in 
the applications was not informative enough to explain the task’s purpose. The availability of written 
language was also deemed important since some of the children could benefit from working with written 
language while working with the technologies, in turn increasing the technologies’ advantages considering 
the school context. 


5. CONCLUSIONS 

The purpose of this study was to present principles related to children’s creativity and strengths, and 
technology’ modifiability and transformability for the establishment of a strength-based technology-enhanced 
learning environment with and for children with ASD and to introduce results on how the project succeeded 
in actualizing the principles in relation to children’s actions in the learning environment. The findings 
indicate that the technology-enhanced learning environment introduced in this paper provided many 
opportunities for facilitating the emergence of potential skills, active participation, and learning of children 
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with autistic features. In addition, the strength-based environment facilitated a chance to see the children 
according to their strengths rather than their challenges and to find diversified ways of supporting their 
learning. The modifiable and transferable technical solutions also facilitated individualized learning and 
teaching, thus increasing the possibility of the children’s inclusion both in the school context and in society. 

As technology plays an increasingly important role in children’s lives in modern societies, children who 
are left out of this process are in danger of being disconnected from peers, cut-off from various opportunities, 
disadvantaged, and unskilled in terms of future work (Montgomery, 2007, p. 210; Vicente and Lopez, 2010). 
It is crucial that technologies are continuously modifiable according to the interests, strengths, and needs of 
children with special needs, including autism. To meet the criteria of children’s various situations, learning 
environments should contain multiple technologies. Every part of a learning environment should be taken 
into account: the people, the technologies, and the pedagogy. 

Technologies should be developed with children with ASD - not just for them. Every child is entitled to 
an opportunity to make choices and affect their environment. It is crucial to establish multiple ways in which 
children with ASD can provide feedback and truly participate in the modification and development of 
technologies. Some recent studies (see Lopez-Mencla et ah, 2010; Nissinen et ah, 2012) indicate that 
participatory evaluation, design, and development of technologies is possible for children with different 
special needs, including autism. An environment with multiple technologies provides a challenging yet 
promising starting point for participatory design. Since technologies interest children with ASD, the aim of 
the near future is to develop technical solutions that facilitate and diversify the children’s inclusion in the 
development of their learning environments. 

The transformability of technological solutions to everyday life contexts also calls for the involvement of 
all participants in the development process. Knowledge of the technologies and skills to use them in various 
ways increase the possibility that school personnel could also use technologies in everyday school contexts. 
As this study’s results indicate, applications have to be easy to use and modify, from the viewpoint of both 
the children and the adults. 

Although results are very promising, there are several limitations in this study. The emphasis of this 
article was on describing the establishment of the learning environment and its technologies and on the 
research’s overall results, instead of focusing on an exact research area. The number of participating children 
was low, which has an effect on the generalizability of the results; however, the project’s learning 
environment works as an experimental environment and the research’s results can be further studied. In this 
phase of the project, the data was limited to a research period of a year and a half, which does not yet meet 
the criteria of a follow-up study. Euture research will give more detailed information about the actions of 
children with ASD and the benefits and limitations of the project’s technology-enhanced learning 
environment. 
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ABSTRACT 

In this project report, we describe the role that a 1:1 smartphone deployment played in transforming the education of P3 
(primary 3/grade 3) students (350-I-) from a didactic/direction instruction pedagogy to an inquiry pedagogy. Based on the 
students’ test scores - Not only did the students learn the prescribed content, but they also developed 21'’* century skills. 
This project builds on a small-scale pilot - and is leading towards a multi-grade/multi-school fully scaled up project. 


KEYWORDS 

Mobile learning, 1:1, smartphones, inquiry curriculum 


1. INTRODUCTION 

The Singapore Ministry of Education’s Masterplan 3 (2010) has called for a change in Singaporean schools. 
The MoE has recognized that the direct instruction pedagogy, with its emphasis on memorization, while 
producing good test takers, is not producing the entrepreneurial, imaginative, innovative thinkers that the 
government feels is important to insure Singapore’s continued growth. 

At Nan Chiau Primary School (NCPS), the administration and the teachers are taking the MP3 very 
seriously. With external funding support, they have implemented a major transformation of their teaching and 
learning practices. As described in the following sections, six transformations were observed in 2012 in grade 
3 at NCPS: 

• Transformation #1 - Pedagogy & Curriculum: 

• Transformation #2 - Technology 

• Transformation #3 - Students Became Self-directed and Collaborative Learners 

• Transformation #4 - Parent Attitude 

• Transformation #5 - Teachers’ Attitudes 

• Transformation #6 - Scaling UP 

And most excitingly, the impact on the grade 3 children is by and large what was desired: 

• Test scores on more traditional measures were not appreciably impacted, negatively or positively. 

• Test scores on 21“‘ century skills were appreciably impacted - in the positive direction. 

In what follows, we describe in more detail each of the 6 transformations. In addition, we describe the 
impact of these transformations on student achievement of the grade 3 (referred to P3 - Primary 3) children. 
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2. TRANSFORMATION #1 - PEDAGOGY & CURRICULUM 

While the topics covered in 2012 in P3 English and science were in alignment with those specified by 
Singapore’s MoE, the pedagogy and the learning activities associated with those topics were in alignment 
with the MoE’s Masterplan 3 (MP3) directives - but in contrast to what typically goes on in a P3 classroom. 
That is, MP3 calls for classrooms to employ an inquiry pedagogy and focus, not only on content, but on two 
key 2E‘ century skills: self-directed learning and collaborative learning. But instead of inquiry, P3 classes 
typically started with direct-instruction, worksheet driven pedagogy. As one youngster, in the HA (High 
Ability) category, commented to a teacher who was asking him questions: “Wait a minute, your job is not ask 
me questions it is to give me answers [to the questions on the worksheets],.” But, by the end of the semester, 
as evidenced by the very high scores on the “Oral” part of the Year-End Test, students were quite adept at 
answering open-ended questions for which there could be no real preparation (because the questions were 
extemporaneous) - while also doing fine on the multiple choice questions for which drill and practice on the 
underlying content had been the standard way to prepare. 


She don't believe that het best 
friend would spread bad 
coments about so she go and 
find evidence to proof it 
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Figure 1. Using Sketchbook to Draw a Picture to 
Develop & Demonstrate Word Understanding - and Then Write a Sentence 

In English , the curriculum that was called “i.m. STELLAR” for ICT. mobilised. STELLAR. (ICT stands 
for the Information and Communications Technologies; mobilized stands for the use of mobile technologies 
- smartphones - being used as an essential tool in the STELLAR (2008, curriculum - the term used by the 
MoE for the P3 curriculum ). 
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In “mobilizing” the STELLAR curriculum - and making it inquiry oriented, not direct-instruction 
oriented, the teachers and curriculum developers drew on two world-recognized pedagogical strategies: 

• P4C, Philosophy for Children, developed by the philosopher Matthew Lipman (1980) with the 
goal of helping children learn how to ask “philosophical questions” - deep questions, questions that 
went to the heart, the assumptions, of the issues under instruction. P4C draws on the Socratic 
method of learning pioneered initially in Plato’s dialogues. Learning how to ask a question and how 
to respond when asked a question - that’s what P4C focused on. 

• 6-steps to Better Vocabulary Instruction, Marzano’s (Marzano, R.J., Pickering, D.J., 2005) 
methods go beyond just memorizing the meanings of words; quite the opposite in fact! Marzano’s 
methods help children understand words by building relationships and links amongst the words, by 
using words in their proper contexts. 



Figure 2. A coherent, cohesive lesson in English that uses 8 different apps - that 3™ graders seem to have no problem 

mastering. 


These techniques were a perfect match for the mobile technology. 

• The children used the Internet that was literally embedded in the palm of their hands to explore 
questions and have verbal discussions to resolve issues. (When a question came up, the children 
would say: “Ask the phone.” Now, the children knew better than to trust everything “the phone” 
said, i.e., the answers returned from the Internet. To foreshadow a later section, we point out here 
that the students using the smartphones with the P4C pedagogy scored exceedingly high on the 
open-ended, oral, question -answering part of the test. Clearly, the P4C helped the children develop 
valuable, 2L‘ century skills! 
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• And, the children had apps on their smartphones that enabled them to use words in drawings, in 
animations, in concepts as well as in textual contexts. This multi-modality opportunity was 
absolutely key - especially for the MA (Mixed Ability) and LA (Low Ability) students. In Figure 1, 
we see drawings that students made to describe words such as evidence and red herring. They used 
illustrations to see if they actually understood the word. After they drew the pictures, they were in a 
better position to give the word its textual meaning. 

• In Figure 2, then, we present an entire lesson where P4C and Marzano are combined into a coherent, 
cohesive series of learning activities. 

In science, curriculum that was used was called “Seamless Learning (SL).” The idea behind SL is that 
learning takes place all the time and everywhere - in school as well as after school; learning is, in effect, 
24/7. Helping your parents learn about the digestive system at home is part of SL, and the mobile technology 
plays a critical role in supporting SL. For example, in teaching their parents about the digestive system, a 
child might create an animation, or the child might work with their parents to collaboratively create a concept 
map with the key terms. 

At its core. Seamless Learning is a flavor of social constructivism, a flavor of Dewey’s “Learn by doing’’ 
pedagogy: 

• “They [teachers] give the pupils something to do, not something to learn; and the doing is of such a 
nature as to demand thinking, or the intentional noting of connections; learning naturally results.” (J. Dewey , 
1916) 

Note too that Marzano’ s pedagogy is all about intentionally building connections amongst the words, 
about understanding how the words are related to each other. Thus, both in English and in science, while the 
particular flavors of pedagogy have their differences, the pedagogies are, at their roots, inquiry-oriented, 
learn-by-doing pedagogies that reinforce each other. 

Just as in English, the mobile technology afforded the students the ability to use multiple modalities - 
multiple media. Here are two children’s understandings of how the digestive system works - rendered in an 
animation that is annotated with text. 

Science is ahead of English in that all 8 classes used Seamless Learning, while English was in its pilot, 
first year. As such. Science took a step further in its curriculum evolution - science tried to create a range of 
curricular activities that differentiated learning, that address explicitly the different needs of students in the 
HA, MA, LA classes. Next year, English will also address the issue of differentiated learning and develop 
materials specifically for the HA, MA, LA classes. 


3. TRANSFORMATION #2: TECHNOLOGY 

The use of mobile devices as an everyday tool for learning is a huge transformation for a school. The school 
is making a sincere commitment to making Internet access as easy as getting a drink of water. While the goal 
is clear - the path to achieving the goal is less so: 

• SingTel continues to throw up administrative issues and its coverage at NCPS is less than 
wonderful, and 

• Nan Chiau’s physical construction projects present challenges to maintaining an always on, 
always available Wi-Ei signal. 

But, since Nan Chiau’s curriculum is so dependent on network connectivity, solutions are being found. 
The MyDesk mobile learning environment has gone through several major changes over the year. 
Currently, the backend that was developed by students at UMich is being replaced by a version created by a 
commercial concern, WizLeam. WizLearn has a robust Learning Management System (LMS) that is already 
in place in a number of Singaporean schools. Thus, scaling up to other schools will be that much easier if the 
cloud-based student artifact repository and teacher portal are the same across a number of schools. On the 
other hand, we will need to transition away from Windows Phone 7 to another platform, e.g., Windows 
Phone 8, Android, iOS. Indeed, at the University of Michigan the transition of the productivity apps in 
MyDesk to WP8 is already underway. 
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More software is needed!! The students and the teachers have virtually insatiable appetites for new 
software. The lesson depicted in Figure 2 already has the 3'^"' graders using at least 8 pieces of software. 
However, as the teachers develop new learning activities, they will want new apps - as will the students. 
Currently, adding new apps is not a straightforward process unfortunately. Given the experience - and the 
demands - of the students and the teachers at Nan Chiau, we are rethinking the architecture of MyDesk. 


4. TRANSFORMATION #3 - STUDENTS BECAME SELF-DIRECTED 
AND COLLABORATIVE LEARNERS 

With an Internet-connected smartphone virtually glued to the palm of their hands, students grew more 
independent and more inquisitive. Teachers observed them asking questions that were not exactly in the 
curriculum - and then pursuing their questions using not only their technology, but conversations with their 
peers. The teachers, over the course of the year, learned how to encourage and nurture this growing 
independence. Teachers encouraged students to see school and learning as 24/7 - use the technology to link 
the abstract ideas explored in the classroom to the concrete, real world instantiations that appear on one’s 
walk home, appear at the dinner table, appear in the mall. 

The theme of the Qualcomm project - WeLearn - has truly become ingrained in the culture of the P3 
students - and teachers. Learning is in the conversation; learning is in the pursuit of understanding. Even at 
P3, even with a small-screened device, learning is all the time, everywhere. 


5. TRANSFORMATION #4 - PARENT ATTITUDE 

At the beginning of the 2012 the parents were nervous about the “new” program in science and English. They 
were concerned that there children were “playing” with the phones and not doing their homework. Mr. Tan, 
the principal, fielded a number of concerned calls from parents. However, over the course of the year, the 
concerns seem to evaporate as the parents saw their children producing interesting artifacts and spending 
considerable time on their devices doing so. The final straw appears to be the test scores at the end of the 
year. Children using the smartphones and the inquiry curriculum performed on par with the students who 
were using the more traditional, direct-instruction curriculum in addition to acquiring all of the, as yet 
unquantifiable skills relating to self-directed learning, collaborative learning and self-expression. 


6. TRANSFORMATION #5 - TEACHERS’ ATTITUDES 

At the beginning of 2012, there was concern that the inquiry pedagogy would be too difficult to implement 
and it would take up additional class time to use the technology. However, over the course of the semester, 
those fears turned out to be groundless. In fact, one teacher said “Teaching is fun now. I know what I am 
supposed to teach, but I don’t know how it will go in class. I have to follow the students’ responses so the 
path is different in each class.” This teacher is talking about using her skills as a teacher in a way that she 
hadn’t been using them when she was enacting a direct instruction pedagogy. The sentiment expressed by 
this one teacher was essentially universal among the English and science teachers. 


7. TRANSFORMATION #6 - SCALING UP 

Because four other principals expressed an interest in transforming their schools too. Nan Chiau will be 
hosting meetings with teachers from other schools during 2013. Teachers from the other primary schools will 
be visiting classes at Nan Chiau in order to see how “it looks.” Teachers learn by watching other teachers. 
The realization that “I can do that” comes from classroom observations. 
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There are a myriad of challenges in scaling up - making the curriculum sensitive to the each school’s 
context, developing professional development programs for the new teachers, creating a robust technological 
infrastructure in the various schools, etc. Those are all barriers to change. But, the biggest barrier - the 
willingness to change in the first place - has been breached. 


8. END OF YEAR TEST SCORES: AN ANALYSIS 

There is one more element to describe: end of year test scores in English and in science: 

English: We did an analysis of the English scores, comparing the children who used smartphones for 
learning vs the children who didn’t. There is a seemingly almost natural comparison group in that 3 classes 
used smartphones and 5 didn’t. But the problem is this: each of the classes was different, thus lumping the 3 
smartphone classes together and comparing them to the 5 non-smartphone classes isn’t statistically “fair.” 
That said, we will do it, just to give a sense of the comparative performance. 

There were four measures on the English language test: Oral, Language, Listening Comprehension, and 
Composition. While there were slight differences - in favor of the traditional, non-smartphone using 
students, there was a whopping difference in the oral skills - in favor of the smartphone using children. 

Simply put: the children using the inquiry curriculum, with the mobile devices, learned how to ask 
and respond to questions. The children using the inquiry curriculum learned how to think on their feet, in 
real time. There are no drill and practice techniques to help children improve their oral skills. The way to 
develop oral skills is to practice them by engaging in conversation, in question asking and in question 
responding. A focus of the curriculum designers in 2013 in English will be to polish and improve the other 
lesson components so that the children will achieve even more in the years following this initial pilot. 

Science: There were 8 science classes. What we compared, then, to see about growth of understanding is 
how the children scored on the SAl test, given in April and the SA2 test, given in November, at the end of 
the school year. We also compared the students’ scores in 2012 with their scores in 201 1 and 2010. Here are 
the key findings. 

The students in P3 are definitely learning a vital 2E' century skill: how to respond to an open-ended 
question. Here is the statistical evidence: 

• The whole cohort (HA, MA, LA) improved more on the Open-Ended (OE) questions than on the 
Multiple-Choice Questions (MCQ). 

• The whole cohort improved significantly on the Open-Ended questions when compared with how they 
did on the OE questions in 2010 and 2011. 

And, it is key that the students are not doing better on OE at the expense of MCQ; while the students are 
not showing significant gains on the MCQ section - they are not showing any significant drop, either. 
Interestingly, as we see below, the gains are coming from the MA and LA groups! 

It appears that the technology-supported, inquiry pedagogy-based learning experiences are significantly 
benefiting the MA and LA groups. The HA group already has very high scores on both the SAl and SA2 
tests and on both sections, MCQ and OE. 

• The MA/LA cohort improved more than the HA cohort due to their high scores on the OE section since 
there was no improvement on their MCQ scores. 


9. CONCLUDING REMARKS 

The project herein described is an education project, not a technology project. The goal is educational 
change; a catalyst for that change is mobile technology. Eirst and foremost the mobile technology empowered 
the students to be independent learners, to “own” their own learning. With a mobile device in the hands of 
each student, 24/7, a student was no longer dependent on the teacher or the textbook as the giver of 
information. Eor example, during a discussion, when a question arose, the children would say: “ask the 
phone” meaning let’s look up the information on the Internet. 
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The ripple effect of this independence was great. As described above, the various transformations were 
now possible: 

• Transformation #1 - Pedagogy & Curriculum: 

• Transformation #2 - Technology 

• Transformation #3 - Students Became Self-directed and Collaborative Learners 

• Transformation #4 - Parent Attitude 

• Transformation #5 - Teachers’ Attitudes 

• Transformation #6 - scale up - is now about to take place as grade 4 is about to undergo change and 
several other schools are looking to adopt (and adapt) Nan Chiau’s educational program for their grade 3 
classes. 
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ABSTRACT 

In this conceptual paper, we propose an heuristic to balance context-specific and generic scaffolding, as well as 
computer-based and teacher scaffolding, during instruction centered on authentic, scientific problems. This paper is novel 
in that many researchers ask a dichotomous question of whether generic or context-specific scaffolding is best, and fail to 
focus on what processes and cognitions each type of scaffolding excels at supporting. To arrive at the heuristic and 
construct the framework, we synthesized research on (a) student challenges gaining and using scientific reasoning 
strategies, and (b) computer-based and teacher scaffolding. 
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1. INTRODUCTION 

When appropriately deployed, scaffolding can play an important role in supporting higher-order thinking 
(Reiser, 2004; Wood et al., 1976). While most researchers agree that both teacher and computer-based 
scaffolding is needed (McNeill and Krajcik, 2009; Saye and Brush, 2002), there is little agreement on how to 
effectively balance these two types of scaffolding (McNeill and Krajcik, 2009). Furthermore, scaffolding can 
be either context-specific or generic, yet little research addresses when and why each type of scaffolding 
should be used (Belland, In press). In this conceptual paper, we propose an heuristic to balance 
context-specific and generic scaffolding, as well as computer-based and teacher scaffolding, during 
instruction centered on authentic, scientific problems. This heuristic was developed through synthesis of 
research on (a) student challenges learning and using scientific reasoning strategies, and (b) computer-based 
and teacher scaffolding. This paper is important because many researchers fail to focus on the processes and 
cognitions each type of scaffolding excels at supporting. Scaffold designers may find this paper useful as 
they design technology-enhanced learning environments and work with teachers to facilitate student use of 
such. 

This paper is organized as such; First, to provide context for the skills that scaffolding supports, we 
discuss authenticity in education. Then, we explore the nature of scientific reasoning, and the extent to which 
it employs context-specific and generic skills. Next, we define and discuss scaffolding and discuss 
research-based strategies for balancing different scaffolding types. Last, we mention remaining questions and 
suggestions for future research. 


2. AUTHENTICITY IN EDUCATION 

2.1 What Tasks have the Potential to he Authentic 

According to canonical authenticity, educators interested in providing authentic experiences simply need to 
determine who the professionals are in the target domain, what they do (both mental and physical processes), 
and have students do the same thing. 
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However, this view is problematic for several reasons. First, recent research indicates that it is unrealistic 
to expect students to carry out the same research that a professional scientist would, because there is too 
much knowledge, skill, and expensive equipment required to do so (Chinn and Malhotra, 2002). But it may 
be reasonable to expect students to carry out somewhat simplified tasks that involve similar epistemic 
processes (Chinn and Malhotra, 2002; Lee and Songer, 2003). Next, science always happens in a cultural 
context (Barab et ah, 2000; Hung and Chen, 2007). This cultural context contains colleagues and the 
theoretical stances they take, and the theoretical stances the individual scientist takes. This influences what 
questions are addressed and how, and how the resulting data are interpreted (Kuhn, 1996; Nersessian, 2008). 
Last, what teachers, students, and professionals see as authentic are not always the same (Barab et ah, 2000; 
Nason and Woodruff, 2003). Students can see activities as authentic if they think the skills being learned can 
be applied in their lives (Barab et ah, 2000). Not all middle school students intend to become scientists when 
they grow up, but they are all citizens who will need to make informed decisions on scientific issues (e.g., 
climate change or local pollution). 

2.2 Manner in which Authenticity is Established 

In the traditional view of authenticity, one simply needed to package what professionals do in the real world, 
bring it back to the classroom, and students would recognize that what they have in front of themselves is 
authentic (Barab et ah, 2000; Rahm et ah, 2003). Two recent ideas have emerged for how to establish 
authenticity. First, by working alongside professional scientists, students interact with scientists as scientists 
interact with each other, and in turn see that they are engaging in authentic science (Hung and Chen, 2007). 
But this approach presents some challenges. The job descriptions of professional scientists usually do not 
involve helping K-12 students as apprentices (Barab et ah, 2000). Also, it is challenging to get professional 
scientists and students in the same place. Second, students may be able to perceive authenticity through 
negotiation of project parameters with other students and teachers (Barab et ah, 2000; Rahm et ah, 2003). 
One issue with this approach is that teachers and students often have misconceptions about how scientific 
ideas are developed and established (Rahm et ah, 2003). This may be addressed through use of augmenting 
persons (i.e., target professionals) who can interact with the class for part of the unit (Hung and Chen, 2007). 


3. SCIENTIFIC REASONING 

Scientific reasoning involves addressing questions about the natural world with data, rational reasoning from 
prior knowledge, or a combination of both (Giere, 1990; Nersessian, 2008; Schunn and Anderson, 1999). It is 
important to consider the roles of data, domain-specific conceptual knowledge, domain-general reasoning 
strategies, and domain-specific reasoning strategies in scientific reasoning. 

3.1 Data in Scientific Reasoning 

To move from explanations of one isolated incident to theories that explain and predict a larger range of 
phenomena, it is important to integrate research findings of many scientists through argumentation (Osborne, 
2010). When constructing arguments, one needs to (a) make a claim that is supported by the preponderance 
of data, (b) present support for the claim, and (c) revisit the claim in light of additional evidence and critiques 
(Ford, 2012; Perelman and Olbrechts-Tyteca, 1958). 

3.2 Domain-specific Conceptual Knowledge in Scientific Reasoning 

Domain-specific conceptual knowledge is important in that it can help scientists determine the relevance of 
and how to interpret particular data (Brand-Gruwel and Stadtler, 2011; Smith, 2002). For example, a 
physician who is trying to diagnose the cause of muscle spasms in a patient's leg would draw on content 
knowledge about physiology and muscle structure in the course of diagnosis. However, the diagnosis strategy 
would share much with that used by a mechanic diagnosing a transmission problem (Smith, 2002). 
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3.3 Domain-general and Domain-specific Skills in Scientific Reasoning 

The idea of domain-general reasoning strategies has received much criticism in recent years, in part due to 
the practice of developing problem solving heuristics from studying people solving well-structured problems 
(e.g., Towers of Hanoi) in laboratory settings (Perkins and Salomon, 1989). Many authors in turn argued that 
most scientific problem solving is domain-specific. However, recent research indicates that scientific 
reasoning cannot be fully explained by domain-specific reasoning strategies (Abd-El-Khalick, 2012; Schunn 
and Anderson, 1999; Smith, 2002). In fact, scientific problem solving often involves a mix of 
domain-specific and generic skills (Abd-El-Khalick, 2012; Jonassen, 2011; Lawson, 2010). Domain-specific 
skills often predominate when scientists encounter problems that are similar to those encountered in the past, 
whereas domain-general skills predominate when scientists encounter novel problems (Roberts, 2007). 

It is unwise to consider thinking skills as general across tasks; rather, thinking skills can be thought of as 
general across domains when engaging in similar tasks (Jonassen, 2011; Smith, 2002). Problem types range 
from the most well-structured (story problems) to the most ill-structured (design problems) (Jonassen, 2011). 
Eor example, troubleshooting a water quality problem and fixing a leaky roof may involve many of the same 
strategies. But troubleshooting problems and designing experiments in water quality analysis likely share few 
reasoning strategies. 

Domain-general strategies include (a) thinking dispositions such as thinking planfully and being curious, 
(b) basic thinking techniques, such as problem solving and evaluation strategies, (c) tool use, such as 
brainstorming, concept mapping, and systems thinking, and (d) process-specific thinking skills, such as 
negotiation (Perkins, 1995). 

Domain-specific knowledge and skills - declarative knowledge, concepts, and problem solving strategies 
- are needed in scientific problem solving. Domain-specific skills include (a) field-specific thinking skills, 
such as the differing ways evidence is used in law versus fine arts, and (b) situation-specific thinking skills 
(Perkins, 1995). Greater command of domain-specific declarative knowledge and concepts makes it easier to 
constrain the problem space (Duncan, 2007). Eor example, when creating a model of a river ecosystem, it is 
helpful to know what insects, fish and plants typically live in the river, what insects the fish eat, what insects 
cut up leaves, and what insects are most sensitive to pollution (Duncan, 2007). 


4. SCAFFOLDING 

4.1 Original Conceptualization 

In its original conceptualization, scaffolding referred to dynamic support provided by a teacher or a parent 
that allowed a child to engage in a task that he/she could not complete without assistance (Belland, In press; 
Wood et ak, 1976). This process involves "(a) enlisting student interest, (b) controlling frustration, (c) 
providing feedback, (d) indicating important task/problem elements to consider, (e) modeling expert 
processes, and (f) questioning" (Belland, In press). Scaffolding today includes three variations - one-to-one, 
computer-based, and peer (Belland, In press) - of which two will be explained in this paper. 

4.2 One-to-one Scaffolding 

One-to-one scaffolding follows the original definition of scaffolding in that it relies on dynamic assessment, 
provision of just the right amount of support, and fading of the support (van de Pol et ak, 2010). But it varies 
in that the typical K-12 classroom setting has about 30 students and one teacher (Belland, 2012). This makes 
one teacher incapable of providing all scaffolding support that her students need (Saye and Brush, 2002). 
Teachers can provide support that is context-specific or generic according to what dynamic assessment 
indicates that the student needs at that point in the unit. In the literature, one-to-one scaffolding most often 
employs questioning, modeling expert processes, and providing feedback (Belland, In press). This works 
well in assessing the clarity of and helping students improve articulated ideas (Jadallah et ak, 2010; Maloch, 
2002). As dynamic assessment indicates that students gained skill, scaffolding support can be reduced until 
students can complete the scaffolded task independently (Collins et ak, 1989; Wood et ak, 1976). 
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4.3 Computer-based Scaffolding 

Computer-based scaffolding can be designed to serve many of the same functions as one-to-one scaffolding. 
However, computer-based scaffolding is not generated on the basis of dynamic assessment, but rather based 
on an analysis of projected student difficulties (Belland, In press). Scaffolding should serve to (a) structure 
tasks through simplification and guidance, and (b) highlight particularly important aspects of the learning 
task. This support can be either generic or context-specific (Reiser, 2004). Such scaffolding often leads to 
desirable outcomes such as evaluation using epistemic terms (Sandoval and Reiser, 2004) and effective 
integration of ideas with groupmates (Gijlers et ah, 2009). 

Generic scaffolds can be helpful in structuring the overall process (Belland, 2010; Reiser, 2004) and 
encouraging students to be reflective in their work (Davis, 2003). For example, while preparing a science 
news story, middle school students were exposed to generic prompts that encouraged them to "stop and 
think" (Davis, 2003, p. 92). Davis (2003) found that high autonomy students performed better with generic 
scaffolds than low-autonomy students. As another example, high school students use the Collaborative 
Concept Mapping Tool to collaboratively create a model of a problem using concept mapping (Gijlers et ak, 
2009). Students who used this tool engaged in more detailed conversations aimed at integrating their ideas 
than students using a context-specific scaffold (Gijlers et ak, 2009). 

Context-specific scaffolds tailor scaffolding support to particular content. For example, 
ExplanationConstructor asks students questions related to the evolution of finches on the Galapagos Islands 
(Sandoval and Reiser, 2004). In the How Do We Make New Stuff from Old Stuff unit, context-specific 
scaffolds gave middle school students step-by-step instructions for what they needed to do (McNeill and 
Krajcik, 2009). Students who received context-specific computer-based scaffolds and generic one-to-one 
scaffolding wrote better scientific explanations than students who received generic computer-based 
scaffolding and context-specific one-to-one scaffolding (McNeill and Krajcik, 2009). 

4.4 Mixing Generic and Context-Specific Scaffolding 

One should not ask whether to use generic or context-specific scaffolding in supporting students' scientific 
reasoning, but rather, one should ask what is the right mix between the two types of scaffolds (Abrami et ak, 
2008). Given that K-12 students need both one-to-one and computer-based scaffolding (McNeill and Krajcik, 
2009; Saye and Brush, 2002), and also strengthening in both domain-specific and domain-general skills 
(Schunn and Anderson, 1999), one can consider having some scaffolding be context-specific and some 
generic. There is limited research on what mix of generic and context-specific scaffolding works best 
(Belland, In press). In the context of the McNeill and Krajcik (2009) study, evidence was found that having 
context-specific computer-based scaffolds and generic one-to-one scaffolding worked the best. But it must be 
noted that the finding is related to one study in one context, and further research is needed in other contexts. 

Factors considered in this section include research on authenticity in education and scientific reasoning, 
and what one-to-one and computer-based, as well as generic and context-specific scaffolding, support best. 

4.4.1 Authenticity in Education Considerations 

One problem with developing context-specific computer-based scaffolding is that it relies to a large extent on 
pre-packaging authenticity. This is because computer-based scaffolding needs to be created prior to student 
engagement with the unit. While one may be able to identify and embed some domain-specific processes by 
which professional scientists engage in science, one would not be able to embed all such processes. Science 
is not a static field, and depending on the perspective that a scientist brings, many reasonable approaches can 
be taken when solving a given problem (Sarkar, 2007). And embedding the social context is very difficult. 
Given that computer-based scaffolds are developed prior to student engagement in the problem, only pre- 
selected strategies could be embedded into computer-based scaffolds. 

Embedding context-specific support in computer-based scaffolds is also limiting in that if particular 
students do not perceive the target problem as authentic, then there are two possibilities: (a) the 
computer-based scaffolds are used and the unit is not perceived as authentic, or (b) a new unit needs to be 
developed that the target students can perceive as authentic and other computer-based scaffolds need to be 
identified or developed. Generic computer-based scaffolds can be used with a variety of units such that a 
problem that target students can perceive as authentic can be selected/developed. 
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4.4.2 Scientific Reasoning Considerations 

To determine what scaffolding can be context-specific and what scaffolding should be domain general, it is 
important to think about the type of domain-general and domain-specific skills involved in scientific problem 
solving. The main domain-specific skills include methods of thinking specific to particular domains and 
contextual modes of thinking (Perkins, 1995). 

But at the same time, students need context-specific scaffolding support, especially to learn contextual 
cues related to the target problem. Such support can be provided through one-to-one scaffolding. With 
one-to-one scaffolding, one does not need to prepackage authenticity, and can give just the right amount of 
context-specific support, as it is needed. Augmenting persons can help the teacher provide one-to-one 
scaffolding for part of the unit (Hung and Chen, 2007). 

4.4.3 Proposed Approach 

One does not need to make all one-to-one scaffolding generic or context-specific, and do the same with 
computer-based scaffolding. Rather, one can consider whether one-to-one or computer-based, and generic or 
context-specific support may be most appropriate for each major area in which students need scaffolding 
support. One can also use augmenting persons - professional scientists - who can provide some one-to-one 
scaffolding support when they visit the classroom (Hung and Chen, 2007). 

To help students learn the cultural and domain contexts of problems, context-specific one-to-one 
scaffolding may be optimal. Computer-based generic scaffolding may be ideal to help students learn about 
the group dynamics by which scientific knowledge is constructed. A mix of context-specific one-to-one and 
generic computer-based scaffolding may be optimal to help students learn scientific reasoning strategies. 

Critical to helping students develop thinking dispositions (e.g., toward clear and deep thinking) is 
assessment and feedback (Halpern and Butler, 2011). Generic one-to-one scaffolding provided by the 
classroom teacher is uniquely positioned to provide assessment and feedback to help students understand and 
move toward such thinking dispositions (Halpern and Butler, 201 1). The classroom teacher can contextualize 
a student's articulation of ideas and dynamically determine what feedback the student needs to improve 
his/her thinking (van de Pol et ak, 2010). Information about thinking dispositions can be provided by generic, 
augmenting, one-to-one scaffolding and generic computer-based scaffolding. A meta-analysis indicated that 
effect sizes were highest when instruction on critical thinking dispositions was delivered in a generic manner 
in the context of a specific course (Abrami et ak, 2008). Critical evaluation of strategies is crucial to the 
development of critical thinking dispositions (Friedel, 2006; Ku and Ho, 2010), but students may not be able 
to provide such feedback by themselves in the form of metacognition (Azevedo, 2005). 

Basic thinking techniques include decision making, problem solving, remembering, and justification 
(Perkins, 1995). It makes sense to use generic computer-based scaffolds because such strategies are generic 
(Schunn and Anderson, 1999), and if any content knowledge needs to be applied along with the strategies, it 
can be learned through other means. This also would allow students to have access to such support when they 
need it, rather than simply when the teacher is available. 

Tool use involves employing strategies such as brainstorming, concept maps, stepwise strategies, and 
graphic organizers (Perkins, 1995). Brainstorming and concept mapping has been successfully supported by 
generic computer-based scaffolding (Cho and Jonassen, 2002; Gijlers et ak, 2009). Following a generic 
approach may be best because these are generic processes that can be used in multiple contexts (Chen et ak, 
2011; Dougherty et ak, 2012). Furthermore, students can be stimulated to uncover relevant concepts for their 
concept maps, for example, through one-to-one scaffolding related to situation-specific thinking techniques. 

Process-specific thinking skills involve utilizing specific thinking systems such as logical, mathematical, 
scientific, or linguistic systems in appropriate fields. To solve mathematical problems or create scientific 
explanations, students might need to use process-specific thinking skills such as formal deduction, 
taxonomies, or probability (Perkins, 1995). These thinking skills can still be applied in many contexts and the 
general approach does not vary much across fields. Therefore, process specific thinking skills are optimally 
supported through generic computer-based scaffolding. 
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Field-specific thinking techniques are specialized forms of thinking dispositions, skills and techniques 
applied within a particular field (Perkins, 1995). Although certain basic thinking techniques or 
process-specific thinking skills are generic in nature, the way to utilize them can vary across different fields. 
For instance, one basic thinking technique - justification - is involved in many fields that call for evidence 
and arguments. However, what counts as evidence can vary by field and professions. Acquiring and utilizing 
field-specific thinking techniques is ideally supported through augmenting, one-to-one scaffolding, since 
augmenting persons can model expert-thinking and behaviors (D. Hung & Chen, 2007). Field-specific 
thinking techniques may be supported through context-specific computer-based scaffolding and classroom 
teacher one-to-one scaffolding (McNeill and Krajcik, 2009; Sandoval and Reiser, 2004). One possible way 
such context-specific support could be embedded into an otherwise generic scaffolding system is to have a 
space where video can be uploaded of an augmenting person. Alternatively, one could provide augmenting 
artifacts (cases, living stories, accounts, and ideas that occur in certain fields or professional communities). 

Although situation-specific thinking techniques often are special cases of general thinking skills, they 
have their own distinctive character and consideration for different situations such as experimental design 
and negotiation. For example, when scientists design experiments, they need to apply and adapt their basic 
generic thinking techniques such as justification, tool use such as concept maps, process-specific thinking 
techniques such as deductive logic, and field-specific thinking techniques to solve the problems they have 
and provide scientific explanations of them. Due to the specificity of situation-specific thinking techniques, 
these techniques can vary within certain fields or professions. Hence, situation-specific thinking techniques 
are also ideally supported through augmenting, one-to-one scaffolding (Hung and Chen, 2007), but may also 
be supported through context-specific computer-based scaffolding (McNeill and Krajcik, 2009) and 
classroom teacher one-to-one scaffolding. As with field-specific thinking techniques, a possible way such 
context-specific support could be embedded into an otherwise generic scaffolding system is to have a space 
where video can be uploaded of an augmenting person or augmenting artifacts could be added. 


5. SUGGESTIONS FOR FUTURE RESEARCH 

A big question is if all scaffolding systems need to incorporate support for each of the six major skills. One 
consideration is the idea of redundancy in scaffolding such that there is a wide range of supports to fit a wide 
range of abilities among students (Puntambekar and Kolodner, 2005). According to this paper, multiple 
forms of scaffolding support diverse skills. More research is needed to see if this model has the potential to 
support student learning effectively. 

It is also unclear how many days augmenting persons would need to come to the unit to afford the 
emergence of authenticity and support the performance and learning of field-specific and situation-specific 
thinking techniques. This is potentially the trickiest form of scaffolding to implement, because working 
professionals do not have unlimited free time to devote to supporting student learning. 

Another question relates to the role of peers in helping students develop scientific reasoning skills. The 
scope of this paper did not allow peer scaffolding to be explored, but peer scaffolding is important to the 
development of higher-order thinking skills (Hakkarainen, 2004). In problem-based learning, group dynamics 
are central to student success (Hung, 2011). A natural question is if there is a way to integrate peer 
scaffolding into the approach outlined above. 


6. CONCLUSION 

As is argued in this paper, it is important for the research community to consider how to balance context- 
specific and generic, as well as teacher and computer-based scaffolding, to effectively support students 
higher-order thinking abilities during instruction centered on authentic scientific problem solving. We 
proposed an heuristic to this end centered on the seven thinking skills coined by Perkins (1995). This 
involves a big shift in the mode of thinking of scaffolding scholars. It is our hope that this paper provides the 
initial push to extend the initial ideas of a distributed cognitive system to include generic and context-specific 
scaffolding in addition to teacher and computer-based scaffolding as well as computer-based and teacher 
scaffolding (Puntambekar and Kolodner, 2005). 
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ABSTRACT 

This study examined the effectiveness of using a Facebook group to increase preservice teachers’ knowledge when one 
was used as a forum to share, answer, and discuss content-related questions in a technology course required for all 
students seeking teacher licensure. Further, it examined the students’ prior use of Facebook groups, how the treatment 
group used the group, and their perspectives of the use of Facebook as an educational tool. The results revealed no 
significant gain in achievement. Almost all participants had prior experience using a Facebook group, and the primary 
purposes of these groups were for organizing events, communication within organizations, communication within classes, 
and lending support to memorials/dedications. Although participation in the group was required and linked to a grade, 
most of the participation was characterized as very low level (i.e., “liking”), with only half supplying the answers to 
questions and about one fifth making comments. Their perspectives on whether Facebook can be used for educational 
purposes were lukewarm, yet they indicated significant change in their perception that Facebook assignments were an 
invasion of privacy. While they perceived the idea to be good, they viewed the expectations for participation as too 
lenient, thus causing lack of in-depth participation. 


KEYWORDS 

Facebook, social networking, social learning, teacher education. 


1. INTRODUCTION 

Pempek, Yermolayeva, and Calvert (2009) reported that social networking sites (SNSs), including Facebook, 
MySpace, and Twitter, have become a common part of everyday life, especially for young adults. These 
web-based services allow the display of information through a profile page (Aydin, 2012; Hew, 2011; 
Pempek et ak, 2009) that may be public or semipublic and allow members to connect socially (boyd & 
Ellison, 2008). Pempek et al. (2009) concluded that social networking sites are rising in popularity and are 
providing a unique way for people to communicate with one another. 

Multiple authors (Cheung, Chiu, & Lee, 2010; Colder, Wilkinson, & Huberman, 2007; Junco, 2011; 
Roblyer, McDaniel, Webb, Herman, & Witty, 2010) concur that the most popular SNS for college students is 
Facebook. Described as an online SNS in which individuals can share personal information and photographs 
as well as connect and communicate with friends (Aydin, 2012; Pempek et ak, 2009), Facebook was created 
by Harvard student Mark Zuckerberg in early 2004 and emerged as a “Harvard-only online social networking 
site” (Hew, 2011, p. 663). It spread to other higher education institutions and opened to the public in 2006 
(Facebook, 2013b). As of March 2013, this SNS reported more than 1.1 billion active users (Facebook, 
2013a). 

Facebook allows members to organize into groups that relate to “personal and professional affiliations, 
which might include educational affiliations, workplaces, interests, and political and religious beliefs” 
(Aydin, 2012, p. 1094). A Facebook group provides the setting for this study, which examined changes in 
achievement as measured by knowledge when used as a forum to distribute, answer, and discuss content- 
related questions in a technology course required for all students seeking teacher licensure. Further, the study 
examined the students’ prior use of Facebook groups, how the treatment group used the group, and their 
perspectives of the use of Facebook as an educational tool. 
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2. LIT REVIEW AND PURPOSE 

2.1 Use of Eacebook by College Students 

Research studies reveal that Facebook users are mostly students (Aydin, 2012). Researchers from the Pew 
Research Centers’ Internet and American Life Project (Jones & Fox, 2009; Lenhart, 2009; Lenhart, Purcell, 
Smith, & Zickuhr, 2010) found that 67-75% of college-aged adults (not necessarily students) use social 
networking sites. Additionally, the EDUCAUSE Center for Applied Research (ECAR) conducted a study 
with 36,950 students who attended 126 American universities and one Canadian university, which 
determined that of the 90% who used social networking sites, 97% reported that they used Eacebook (Smith 
& Caruso, 2010). This study found further that Eacebook use by college students increased from 89% in 
2008 to 97% in 2010. The growth of Eacebook and its popularity among college students suggests that its 
use in educational contexts can be “a potentially powerful idea” (Mazman & Usluel, 2010, p. 444). While 
the use of Eacebook by college students is high, researchers have reported harmful effects associated with its 
use. 

2.2 Harmful Effects Associated with using Eacebook 

One harmful consequence associated with using Eacebook is time. Junco (2011) established that students 
devote a lot of time daily to the social networking site, and Kirschner & Karpinski (2010) suggested that this 
time allocation can have a negative impact on academic performance, leading to lower GPAs and less time 
spent on academic work. Other detrimental effects that have surfaced through research studies include 
exhibiting inappropriate behaviors (Butler, 2010), abuse and cyberbullying (Cantanzaro, 2011; Siegel, 2010), 
and issues relating to privacy and friendship (Wihbey, 2010). Others have found evidence that college 
students post inappropriate information and photos to their pages (Olson et. al, 2009; Steinbrecher & Hart, 
2012), which can result in unexpected consequences such as job loss. 

Another drawback to using Facebook for educational purposes is students’ perceptions of its use. 
Students consider communicating on Facebook to be fun rather than serious (Lewis & West, 2009). 
Dahlstrom, de Boor, Grunwald, and Vockley (201 1), in an ECAR study with 3,000 students, found that more 
than half (53%) perceived the academic usefulness of Eacebook to be limited or nonexistent. Other 
researchers (Madge, Meek, Wellens, & Hooley, 2009; Authors [in press]) agree that students’ perceptions of 
educational work differ greatly from their notions of the purpose of Eacebook. In these studies, students 
pointed out that Eacebook is a social networking site to be used for social purposes rather than for 
schoolwork. Consequently, students intentionally establish boundaries to keep the two purposes separate. 

2.3 Positive Effects Associated with using Eacebook in Educational Contexts 

While there are harmful effects associated with the use of Eacebook, some studies indicate that that Eacebook 
can positively affect communication and the involvement of students. Eacebook can positively influence 
educational contexts by providing a way for students to contact and communicate with fellow classmates and 
instructors about course assignments or group projects (Aydin, 2012; Hew, 201 1). Other studies indicate that 
Eacebook expands Internet access (Manzo, 2009), assists with the collapse of previous borders and barriers 
(Schaffhauser, 2009), and increases involvement (Heiberger & Harper, 2008) and student engagement 
(Junco, 2011; Junco & Cole-Avent, 2008). Greenhow (2009) found Eacebook to be a learning space for new 
literacy practices, and Roblyer et al. (2010) suggested that the social learning tool is a resource to support 
students’ communication and collaboration with faculty. Additionally, Schroeder and Greenbowe (2009) 
compared students’ use of Facebook groups with their use of discussion forums in a chemistry class and 
discovered that the students used Facebook more enthusiastically than the forums. 

Wang, Woo, Quek, Yang, and Liu (2012) used a Facebook group as a substitute learning management 
system (LMS), which allowed the posting of announcements, sharing of ideas and resources, and 
implementing online discussions with success; yet, document support (PPT and PDF) was limited and a 
meaningful structure for the organization of discussions was lacking. A recent study (O’Bannon, Britt, and 
Beard 2013) that involved 96 preservice teachers who used a Facebook group to discuss course content 
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showed a significant increase in test scores from pretest to posttest. Further, these preservice teachers liked 
that the group activity resulted in a study guide for exams and that it provided easy access for communication 
and collaboration with other students. 

Despite the fact that Facebook is a social networking site that most college students frequent to facilitate 
social relationships (Madge et ah, 2009; Pempek et ah, 2009), and it has been found to have effective 
collaborative academic potential (Mason, 2006; Selwyn, 2009) and positively affect classroom practice, 
student involvement (Aydin, 2012), and student engagement (Junco, 2011), we know little about how or if it 
can be used to improve learning. 

2.4 Purpose of the Study 

Considering the assertion that most college students use Facebook (Dahlstrom et ah, 2011; Smith & Caruso, 
2010) and the lack of evidence that it can be used to improve learning (Flew, 2011; Junco, 2011) ongoing 
research on Facebook in this area is needed. Our intention is to contribute information to the literature in this 
area, building on a previous study (O’Bannon et al, 2013) by the addition of a control group to determine 
how achievement is affected by the use of a Facebook group. 

The purpose of this study was to examine changes in achievement when a Facebook group was used as a 
forum to share, answer, and discuss content -related questions in a technology course required for all students 
seeking teacher licensure. The research questions that guided the study are as follows: 

1. What effect does using a Facebook group as a forum to share, answer, and discuss content -related 
questions have on achievement, as measured by change in knowledge versus no discussion. 

2. What is preservice teachers’ prior use of Facebook groups? 

3. How does the treatment group use a course -specific Facebook group? 

4. What are the perspectives of the treatment group on the use of Facebook as an educational tool? 


3. METHODOLOGY 

3.1 Participants 

Ninety-seven preservice teachers who were enrolled in five sections of the required technology course were 
invited to participate in the study. Of these students, 90 (96%) completed the study. The mean age of the 
participants was 23.47. Seventy-four (82%) of the participants were female, and 16 (18%) were male. 
Ninety-four (94%) of the participants were White, four (4%) were Black, one (1%) was Asian, and one (1%) 
was Hispanic. 

The five sections were randomly assigned to either the control group (CG) or the treatment (Facebook) 
group (FG). Within the CG, 37 (93%) of the students who were enrolled agreed to participate. Thirty (81%) 
were female, and seven (19%) were male. Within the FG, 53 (96%) of the students who were enrolled 
accepted the invitation to participate in the study. Forty-four (83%) were female, and 9 (17%) were male. All 
participants had a Facebook account prior to entering the class. 

3.2 Data Sources 

Guided by the recommendations of Creswell (2009), this study used a mixed-methods design, involving both 
quantitative and qualitative data collection/analysis to provide a comprehensive view of the data. Pre/post 
exam scores, online survey responses, and Facebook group “wall posts” provided the data used in this study. 

The scores from the pre/post exams were used to answer the first research question. The pre/post exams 
that covered selected course content consisted of 50 multiple -choice questions that were developed by the 
authors of the course textbook, one of whom is the first author of this study, and adapted by the instructional 
team. 

Content validity was established for the exam by using experts (n = 6) in the field of educational 
technology. Additionally, an expert in assessment techniques reviewed the survey and made suggestions for 
changes in some questions. Preservice teachers (n = 300) in the technology course took the exam in past 
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semesters, and questions have been revised for clarity based on their requests. Most questions were retained 
(n = 45), and five were revised as suggested by the experts and students to better communicate the questions. 
None were eliminated. 

The exams were administered through the assessment feature in Blackboard and were given prior to and 
following the study. Pre/post online surveys were used to answer research questions two and four, and 
analysis of the wall posts was used to answer the third research question. 

The first survey was developed by the researchers and was administered at the beginning of the semester. 
This survey consisted of a mix of single-answer, yes/no, and Likert-scaled questions using 5-point scales. 
This survey was used to determine whether the participants had Facebook accounts, their prior experience as 
members of a Facebook group, and their perceptions of using Facebook for educational purposes. 
Specifically, researchers asked participants if using Facebook would be convenient, helpful for their learning, 
or an invasion of their privacy and whether they considered Facebook to be used for personal/social 
interaction rather than for education. Finally, participants were asked to supply demographic data. 

The second survey was developed by the researchers and was administered to the treatment group at the 
close of the semester. This survey consisted of open-response questions and Likert-scaled questions using 
5-point scales. The scaled questions were used to compare, over time, the participants’ perceptions of using 
the Facebook group for educational purposes. The open-response questions that were gathered extended data 
on the participants’ perceptions of using Facebook for educational purposes as well as data relating to the 
management of the Facebook group, including reactions to an avatar that posted the questions, the pacing of 
questions, the notifications that were sent by Facebook each time a post was made, and other thoughts. The 
online surveys were administered by an on-campus research support center. Links to the surveys were 
provided at the course website. 

The “wall posts” within the Facebook group were used to examine how the treatment group used the 
group and the level of their participation in the group. Questions related to course content were posted daily 
on the wall of the Facebook group for 5 weeks. FG members responded to the questions by contributing 
answers, “likes,” or comments (see detailed explanation in the Procedures section below). These “wall posts” 
were analyzed for quantity, type, and content. 

3.3 Data Collection and Analysis 

In each class section, a neutral party distributed the information sheets, explained the study, and invited the 
students to participate. Participation in the research was voluntary with options for withdrawal at any time. 
The instructors were unaware of the identities of participants. The survey data was anonymous, managed by a 
campus research support center, and analyzed after the completion of the semester. 

The pretest exam was administered at the beginning of the semester and provided baseline scores; the 
posttest was administered at the close of the semester. The online strrveys (see Appendix B) were 
administered at the beginning and end of the study. Following the calculation of the survey results, the data 
were analyzed using SPSS statistical analysis software, and appropriate statistical tests were administered. 

The researchers recorded the number and type of contributions on “the wall” weekly. At the conclusion of 
the study, the data were analyzed using SPSS statistical analysis software and frequencies were run. The 
content located on the wall and the open-response questions on the survey were analyzed further using the 
qualitative method described by Bogdan and Biklen (2006). This approach begins with organizing the 
materials into manageable sections followed by taking “long, undisturbed periods” (p. 185) to read through 
the entire collection of data. While reading through the material, a list of preliminary codes is created, after 
which the material is reread, and a formal coding list is created. Next, a final read-through is completed with 
the formal coding list, and the data is assigned a code. Finally, the data is reviewed and tested to ensure that 
the codes fit the data, modifying the list as needed. 
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4. RESULTS 

4.1 Effect on Achievement 

The first research question examined how achievement, based on change in knowledge, was affected as a 
result of using a Facebook group to discuss content-specific questions in a technology course versus no 
discussion. A Repeated Measures ANOVA was administered to compare pre/post exam scores to determine 
if changes in achievement differed for control/treatment groups. The interaction was not significant, [F(l, 
88)= .658, p = .420]. Data for the pretest M = 26.6 and posttest M = 40.3 revealed a significant improvement 
over time, [F(\, 88) = 625.505, p < .001]. 

4.2 Prior Use of a Facebook Group 

The second research question examined the preservice teachers’ prior use of Facebook groups. Almost all (85 
or 94.4%) had participated as a member of a Facebook group. They were asked to identify the types of 
groups in which they participated. The data revealed that, contingent on the purpose of the group, some were 
ongoing, while others were limited in duration. The greatest number (69 or 81.1%) stated that they were 
members of groups created to organize an event, such as a wedding, birthday, party, or trip. More than half 
(58 or 64.4%) reported that they were members of groups aligned with organizations, such as sororities, 
fraternities, or clubs. Almost as many (55 or 61.1%) reported membership in a group related to academics, 
and a third (30 or 33.3%) reported that they were members of groups with a memorial or dedication purpose. 

4.3 Use of a Course-specific Facebook Group 

The third research question examined how the treatment group (FG) used the group in Facebook. The FG 
was required to participate in the activity, and their participation was linked to a grade, which was equivalent 
to 5% of the semester grade. Each member of the FG was expected to make five contributions per chapter, 
and the students could choose the type of contribution they wanted to make. The researchers recorded the 
number and type (answers, likes, comments, other) of contributions made weekly by individual members. 

4.3.1 Answers 

Twenty-seven (50.9%) of the group members provided 53 answers or partial answers to the 25 questions. 
Almost as many (26 or 49.1%) did not contribute answers. The number of answers provided by the 27 
individual members varied, with most (16 or 29.6%) contributing one answer, six (11.1%) contributing two 
answers, two (3.7%) contributing three answers, and three members (5.6%) posting more than three answers. 
The structure of the answers ranged from very short and concise to longer statements, at times containing 
page numbers and/or direct quotes from the course textbook. There were, at times, multiple answers to a 
question. This occurred when incorrect answers were posted and someone corrected the answer or when 
answers were posted that were inadequate or did not fully respond to the question. 

4.3.2 Comments 

Nine (17%) members posted 11 comments to the group, but most (44 or 83%) did not contribute comments. 
The comments provided links to additional information and, typically, were accompanied by statements 
about how the information could be used. Some were accompanied by helpful text such as, “This site may be 
useful,” or “Here is a list of some examples of graphic organizers that can be printed and used in the 
classroom!”. 

4.3.3 Likes 

Most of the contributions (1,239 or 95%) posted by the members of the FG were classified as “liking.” All 
group members participated in this manner. However, “liking” was the only way that 23 (43.5%) members of 
the group participated. Members “liked” both answers and comments, and occasionally some participants 
posted multiple “likes,” with some “liking” up to six contributions in one day. Sometimes a member would 
make all of their contributions in one day and not return to the group until the following chapter; however, 
this occurred with only three students. 
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4.3.4 Other Posts 

During the activity period, some participants posted remarks to questions that did not fall into any of the 
three main types of contributions (answers, comments, or likes). These posts focused primarily on what 
would be considered the social side of Facebook. For example, one participant posted, “Section 005 
Represent!!!” after a fellow elassmate answered a question correctly. This slang use of “represent” means to 
take pride in one's group and was used multiple times by students in one of the course sections. Another 
responded, “#LIVE00Five.” The comment, a takeoff on a local news show (“Live at 5”), has a hashtag, 
commonly used on Twitter to trace topics and search for information or tweets related to a topic. In this case, 
the participants in one class section, which meets at 5 p.m., used this to represent their class (see Appendix 
E). Posts of this sort occurred randomly throughout the study period. 

4.3.5 Perspectives on the Use of Facebook for Educational Purposes 

The last research question used a 5-point Likert scale (1 = SD; 5 = SA) to examine perceptions of using 
Facebook for educational purposes. Participant responses were mixed, but more agreed than disagreed that a 
Facebook group could be used effectively for educational purposes M = 3.36; SD = .982. 

In addition, four questions that appeared on the first and second surveys were compared to determine 
change over time. Overall, the data showed no significant change in participants’ perceptions from pretest to 
posttest, with the exception of their perceptions regarding the educational use of Facebook as being an 
invasion of their privacy. 

Specifically, they were asked if using Facebook for educational purposes was convenient. Responses 
were mixed, but more agreed than disagreed that using Facebook for educational purposes was convenient. 
Results on the first survey revealed M = 3.32; SD = 1.123, while on the second, the data revealed M = 3.34; 
SD = 1.055. Results of a paired f-test (f = -.110, df = 52, p = .913) indicated that students did not change 
their perceptions of convenience from pretest to posttest significantly. 

The participants were also asked if they thought that Facebook would be/was useful for their learning. 
Responses were mixed and somewhat neutral. Results on the first survey indicated that M = 3.00; SD = 
1.038, while on the second, the data revealed that M = 3.19; SD = .982. Results of a paired f-test (f = 1.218, 
df = 52, p = .229) indicated that there was no significant change in their perceptions of Facebook as being 
useful to their learning. 

Additionally, the participants were asked if they considered Facebook to be a personal/social media tool 
rather than an educational tool. Again, responses were mixed, but more agreed than disagreed that Facebook 
was personal/social rather than educational. Results on the first survey showed that M = 3.26; SD = .902, 
while on the second, the data revealed that M = 3.34; SD = 1.055. Results of a paired f-test (f = .438, df= 52, 
p = .663) indicated that the participants did not change their perceptions from pretest to posttest significantly. 

Finally, the participants were asked if they thought that using Facebook would/did invade their privacy. 
Results on the first survey revealed that M = 3.19; SD = 1.057, while on the second, the data showed that M = 
2.68; SD = .996. Results of a paired f-test (f = 3.214, df = 52, p = .002) indicated a significant change in 
perceptions from pretest to posttest; the students were less concerned about the use of Facebook becoming an 
invasion of privacy after participation. 


5. CONCLUSION 

The current study, the first of its kind in the literature on social learning, suggests that using a Facebook 
group has benefits but was not useful for increasing achievement. Yet, the study also revealed flaws in the 
design with respect to the expectations for contributions. Would a change in these expectations yield a 
different outcome? Additional research should be conducted to determine whether a change in design could 
influence achievement. 

Some students are open to the use of Facebook for educational purposes while others are not, which is a 
deterrent to participation. Students must consider the activity valuable, and some accountability must be 
attached to participation, which should be structured to require more in-depth contributions. 
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While some students are quite willing to share their Facebook accounts for educational purposes, others 
are very offended at the suggestion of using it in that way. Encouraging students to view Facebook as another 
avenue for educational context is going to take time. Students must be comfortable with using this social 
networking tool for educational purposes and accept Facebook as an instructional method for it to be 
effective. To facilitate this comfort level, instructors should consider using Facebook in their instructional 
practices. In addition, activities should be planned carefully to enhance student buy-in. 

Although a growing number of articles on the use of Facebook are appearing in the literature, there 
remains a dearth of empirical studies on the effect of this social network on achievement. While the value of 
this vastly popular tool remains unknown and seems insignificant at this point, researchers should conduct 
studies that build on the findings in this study, perform experimental research, and evaluate how Facebook 
can be used effectively, if at all, in education. 
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ABSTRACT 

The rise of digital game and simulation-based learning applications has led to new approaches in educational 
measurement that take account of patterns in time, high resolution paths of action, and clusters of virtual performance 
artifacts. The new approaches, which depart from traditional statistical analyses, include data mining, machine learning, 
and symbolic regression. This article briefly describes the context, methods and broad findings from two game-based 
analyses and describes key explanatory constructs use to make claims about the users, as well as the implications for 
design of digital game-based learning and assessment applications. 


KEYWORDS 

Virtual performance assessment, learning analytics, game-based psychometrics, data mining, machine learning, 
simulation 


1. INTRODUCTION 

The growth of digital game and simulation-based learning and assessment applications has given rise to new 
considerations about how to make sense of what a user knows and can do based on an analysis of interaction 
log files. The log files are typically a time-stamped record of all the actions taken by the user in the digital 
space, so they often provide a high-resolution view of the user’s performance over time. The data files can 
become quite large in comparison to typical educational measurements. For example, it is not uncommon to 
have thousands of records for a single user’ s thirty minutes of virtual performance interaction, compared with 
a dozens or perhaps a hundred responses from a thirty-minute multiple-choice “test.” Several recently edited 
books have begun to bring together findings from researchers who are grappling with the issues of time, 
sequence, action relevancy, big-data pattern recognition, grain size and resolution, overlapping patterns, 
levels of meaning and other intriguing challenges (Ifenthaler, Eseryel, & Ge, 2012; Mayrath, Clarke-Midura, 
& Robinson, 2011; Tobias & Fletcher, 2011). New reports from exploratory research can offer additional 
insight and help lead to ideas that may prove useful for a synthesis of methods that are emerging to deal with 
the data from interactive digital learning applications. 

This article briefly describes the context, methods and broad findings from two game-based analyses and 
provides an abstract of key explanatory constructs utilized to make claims about the users, as well as the 
implications for the design and measurement of digital game-based learning and assessment applications. 


2. LOG FILES AND LEARNING ANALYTICS 

The data for the analyses described here comes from two virtual performance assessments (VPAs) developed 
by the Virtual Assessment Project at the Harvard Graduate School of Education. The VPAs assessed middle 
school students’ abilities to design a scientific investigations and construct a causal explanation (Clarke- 
Midura, Dede, & Norton, 2011). The assessments were designed in the Unity game engine 
(http;//unity3d.com/) and have the look and feel of a videogame (Eigure 1). 
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Figure 1. Screen shots of the Virtual Performance Assessments (VP A) 


The assessments start out with one of two problems that students must solve: Why is there a frog with six 
legs? What is causing a population of bees to die? Students walk around the virtual environment and visit 
farms, talk to farmers, collect data, test the data in the lab, and conduct research until they have gathered 
enough evidence to support a claim that allows them to identify the causal factor. 

Every action by every user (e.g. opening a page, saving a note) was time-stamped as an event. The data 
from pilots of the assessments used in the analysis reported here consisted of 1987 users (423616 event 
records) in the frog assessment and 1958 users (396863 event records) in the bee assessment. The data 
examined in this analysis included the raw event data (up to when they make their final claim about the 
problem) and the scored constructs: designing a causal explanation and designing a scientific investigation. 
The scored data was scored using a rubric generated by researchers. The scored data was stored in a file that 
contained demographic information about students (age, gender, class, teacher) as well as their starting 
prediction for the cause of the problem. 

Designing causal explanation is defined as the student’s ability to support their claim or conclusion with 
evidence. The measure of students’ ability to design a causal explanation (DCE) was operationalized through 
assigning points based on whether the evidence they provided supported the claim they made. Students were 
first asked to identify data that was evidence based on what they collected in their backpack and the tests they 
conducted. They were then allowed to choose from all possible data in the virtual environment, to give 
students who may not have collected all the necessary data a chance to support their claim with evidence. 
Then the student indicated for each piece of data whether or not it was evidence for their claim/conclusion, as 
well as identifying which farm was causing the problem. Most of the evidence and the final conclusion or 
claim were scored on a scale of (3, 2, 1, or 0 points). A backpack of objects populated by the student 
contained up to 5 pieces of data, each worth (3, 2, 1, or 0 points). Overall, DCE is scaled between 0 and 24. 

Designing a scientific investigation (DSI) is defined as the student’s ability to carry out an investigation 
to gather evidence to support their claim. The measure of students’ DSI ability was operationalized through 
assigning points based on whether they conducted tests in the labs, used controls, conducted multiple 
samples, and reviewed informational research on the causal factors. These processes were scored 
dichotomously, if the students performed the action they were awarded a point. If they did not, they received 
a 0. Overall, DSI is scaled between 0 and 24. This construct was an attempt by the researchers to turn student 
investigative processes captured in the log data into products. 

The raw event data contained the time-stamped actions students took from the moment they logged in 
until they were ready to make their final claim, which is called the “event file.” The event file had multiple 
records per user based on the number of events triggered by the user during a single testing session, and 
contained fields including the time-stamp of each event, a code for the zone, action and object of each event 
(i.e. where in the application, using what interaction method, and on what objects associated with each 
event), and the results of in-world interactions that produced a result. Eor example, if the user conducted a 
blood test, there might be five testing results showing which tests were performed on frogs and the result of 
each test. Similar data were available for the bee assessment. 

The purpose of the analysis was to search for patterns of action that might relate to the performance of the 
user correlated with the student’s final claim. Could the log and score data tell us about the user’s 
performance? Ultimately can performance in a virtual performance assessment replace performance on a 
test? Additional questions included: 
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• Is there a relationship between overall duration and score level? 

• Were there performance differences that differed by gender, age, and grade? 

• Was there a relationship between someone’s prediction at the beginning of the assessment and 
their claim at the end? 

• Did students have different patterns of behavior and resource utilization that were predictive of 
their claim? 

• Were patterns of behavior and resource utilization related to their predictions? 


3. TOOLS AND METHODS 

Software tools used in the analyses included Excel, Weka, Eureqa and GraphViz (Table 1). Excel pivot tables 
were used to explore counts and cross-tab relationships among variables. Erom the pivot tables, subsets of 
data were exported to Weka or Eureqa depending on whether the goal of the exploration was data mining 
with cluster methods or symbolic regression. Weka was used to visually inspect data relationships, classify 
datasets, and discover clusters and association rules. Eureqa was used to conduct symbolic regression 
searches for mathematical expressions that could best capture the dynamic relationships among the variables 
under study. GraphViz was used to create network digraphs of the association rules found for subgroups and 
the population as a whole. At the end of this article is a reflection on the strengths and weakness of each of 
these tools and their relationship to the overall analysis. 


Table 1. Tools Used in Game-Based Analysis 


Software 

URL 

Uses 

Excel 

http://office.microsoft.com/en-us/excel/ 

Counts, Pivot Tables 

Weka 

http ://www.cs. Waikato, ac .nz/ml/ weka/ 

Data mining with machine learning 
algorithms 

Eureqa 

http://creativemachines.cornell.edu/eureqa 

Symbolic Regression 

GraphViz 

http://www.graphviz. org/ 

Network graphs 


In this section, examples of the various methods employed are offered as brief introductions to the 
approaches and the primary purpose for selecting each one. The goal here is to set the stage for commenting 
on the psychometric implications by giving an overview of the resulting information obtained with each 
method and its potential relationship to understanding what a user knows and can do being inferred from the 
log file of a virtual performance assessment. 

3.1 Symbolic Regression 

To answer the question about whether duration of performance was related to score, a traditional approach 
might be to seek a correlation and explain the shape of the data from the point of view of the population as a 
whole. In contrast, the symbolic regression method using Eureqa (Schmidt & Lipson, 2009) was selected to 
attempt to obtain a detailed mathematical expression that would be predictive for any individual score given 
the user’s duration in the digital assessment (or vice versa). Correlation in this case is used as a criterion for 
the fit of the discovered equation. To preprocess the data, information on duration and total score was 
smoothed and normalized, and outliers were removed (Eigure 2). Note the cyclic data relationship as the total 
score increases from left to right; this cyclic aspect in the data is due to the nature of the time-stamp, which 
uses modulo math (e.g. the 13* hour resets the hour clock to 1, the 61“ second resets the seconds clock to 1, 
the 61“ minute is an increase of 1 hour, with a resulting zeroing out of the variable and thus a cycle). The 
zero point on the horizontal is the comparable means of the two variables after normalization. A search was 
performed until equations converged, with Eureqa’ s default settings for error (squared error) and simple 
arithmetic expressions (basic operations plus trigonometric building blocks). The selected solution (Eq 1) on 
the Pareto Curve had r^2 = .72 and correlation of .85 (Eigure 3 lower right hand comer). The Pareto Curve 
represents the trade-off in efficiency between error and complexity: the less complex the mathematical 
expression, the higher the error and vice versa. This example of the use of Eureqa provides evidence for the 
finding that complex nonlinear relationships can be discovered via symbolic regression. 
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Eq 1; total_score = 49. 72*duration/(0. 01552 + duration) 



Figure 2. Pre-processing Visualization in Eureqa Showing Smoothed and Normalized Data for Duration and Total Score. 



Figure 3. Relationship of Total Score to Duration: Solution on the Pareto Curve 


3.2 Counts 

To characterize the relationship of prediction to claim, the next example shows the use of the Pivot Table in 
Excel to display a count of the unique occurrences of users (by student ID) in a cross-tab matrix of prediction 
versus claim (Table 2). The Pivot Table automates selected mathematical and string operations on variables 
and arranges the results in a matrix that allows quick exploration of the data. 


Table 2. Unique student ID matrix of counts for prediction and claims in A2 


Count of student_id 

claim_id 



prediction 

aliens 


mutation 

parasites 

pesticides 

pollution 

Total 




7 

12 

14 

10 

43 

aliens 


21 

7 

11 

21 

12 

72 

dunno 


18 

50 

270 

310 

150 

798 

mutation 


5 

126 

168 

209 

103 

611 

parasites 



5 

66 

40 

9 

120 

pesticides 



1 

35 

95 

30 

161 

pollution 


1 

3 

26 

65 

85 

180 

Total 


45 

199 

588 

754 

399 

1985 


Note that 72 students predicted “aliens” as the cause of the unusual frogs (the total of the aliens row), but 
only 21 of those offered “aliens” as their claim (the intersection of the row with the aliens claim column). 
The count table provides the basis for empirical probabilities based on the ratios of students located at the 
intersection of prediction and claim choices. In a similar population of middle school students, we would 
expect that 754/1985 or 38% will likely claim “pesticides.” We can also see that only 8% predicted that result 
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at the beginning of the assessment, so a significant portion of the test takers arrived at this conclusion (and all 
other conclusions) after interactions in the virtual assessment. This indicates that the assessment might be 
educative and that user actions might give clues to a user’s thought processes. 

3.3 Rule Discovery with Machine Learning 

As an example of rule discovery with the aid of machine learning algorithms followed by network analysis, a 
search of the frog assessment data found the ten best association rules using the “Apriori” algorithm in 
WEKA (Witten & Frank, 2005). This algorithm seeks to find patterns in an exhaustive search of the data, 
which can then be used to produce a rule network for the population. Confidence levels are interpreted as the 
probability of finding the noted association rule within this population. 

3.4 Network Analysis 

A digraph was then created with GraphViz (http://www.graphviz.org/) based on the ten best association 
rules, which depicts the network of relationships in the data (Figure 4). A digraph is a “directed graph” where 
the edge from one node to another has a directional meaning - as in causality or implication. An association 
rule network has directionality if there is not a second rule pointing back from a second node to the first. For 
example rule one points from research_3 to research_l with high confidence, but there is no rule pointing 
from research_l back to research_3 within the top ten rules, so the digraph captures the one-way relationship 
and implies that there is more than an association between these two nodes in this particular direction. Such 
would be the case, for example, if many other users scored a “one” on research_l but then never interacted 
with research_3. Experts familiar with the structure of the virtual performance assessment have to validate 
whether the one-way causal or implicative relationship is appropriate for the structure of the virtual space. 



Figure 4. Digraph of Best Association Rules for the Total Population 

The rule and network analysis of the frog assessment led to the observation that subgroups that did not 
have a structure of scientific investigation similar to Figure 4 were more likely to have missed important 
evidence and reached a weaker conclusion. If this information were used during the assessment to re-direct 
students to important evidence, then the digital experience would potentially be formative for developing 
their abilities to design scientific investigations. 

3.5 Cluster Analysis 

As a final example of analysis methods, we used cluster analysis to find out if there was a relationship 
between salient moves and clusters formed from all other data in the record. Salient moves, which had been 
determined by an expert panel, were identified as part of the conceptual framework of the assessment. Each 
salient move counted as “1” in this analysis, which searched for the number of such moves in relationship to 
claim and the user’s closest cluster using all data in the record. Closeness was determined using the 
“Expectation Maximization” algorithm (Dempster, Faird, & Rubin, 1977). Clusters mapped closely to 
claims, but were more complex, because they were formed from all available data. For example, students 
who shared similar search and resource utilization strategies might be clustered together, even though they 
reached different conclusions about the data and made different claims (Figure 4). 
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Figure 5. Number of Salient Moves Vs Cluster Membership 

It is clear from Figure 5 that cluster 2 used far fewer salient strategies than others, indicating that above a 
certain number of salient moves, we can predict which cluster a student is NOT a member of, narrow down 
to the remaining groups and use group probability distributions to estimate other aspects of the student’s 
performance such as total score and final claim. 

The above examples of symbolic regression, counts establishing empirical prior probabilities, 
visualizations, rule discovery, network structure, and cluster analysis methods illustrate some of the new 
array of tools used in game-based data analysis. In the following sections, a comparison of the methods is 
presented followed by a summary of main findings and thoughts concerning the design of game-based data 
collection for educational measurement. 


4. EXPLANATORY CONSTRUCTS AND REFLECTIONS 

This section briefly highlights the main strengths and weaknesses of the analysis approaches when applied to 
log data collected from a virtual performance assessment. Overall, the strengths of all the above methods 
come from their use in model building contrasted with hypothesis testing and traditional statistical testing. 
The methods are useful when the questions about data are open-ended and ill-structured; more along the lines 
of “what have we got here?” than “to what extent is there an impact?” 

Symbolic regression (Schmidt & Lipson, 2009) discovers sets of mathematical expressions that capture 
the dynamics and structures in data, but leaves the decision to the user concerning which expression is best 
for a particular purpose. The expressions are arrayed from most simple with highest error to most complex 
with least error, and there is a danger that if an analyst chooses complexity and low error, and performs no 
other tests or explorations, then an “over-fit” expression will be the result. To ameliorate that threat, cross- 
validation methods are used; random subsets of the data are used to train as well as test the fitness of the 
solution. The method is most naturally used with continuous quantitative data, but additional methods can be 
applied to deal with qualitative data. Groups of such expressions can lead to rule sets, network 
representations and analysis. 

The discovery of association rules among qualitative data can also lead to network representations and 
analysis (Han, Cheng, Xin, & Yan, 2007). Algorithms in data mining toolsets perform exhaustive searches 
and optimization routines that result in a descriptive and associative rule set (compared to the mathematical 
rule set of the symbolic regression method). Such an associative rule set, when considered with the 
confidence of the rule, can elucidate the hierarchal as well as temporal structure (Campanharo, Sirer, 
Malmgren, Ramos, & Amaral, 2011) of the relationships in a virtual performance assessment created by the 
paths of multiple users traversing the space and utilizing resources. A limitation of this method is that it is 
used solely with qualitative data, so continuous data would need to be quantized (Miles & Huberman, 1994) 
before applying the methods. Log data of a qualitative nature does not need to be coded into numeric bins, as 
is the case when using SPSS methods. 
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Visualization methods have been traditionally thought of as the display of findings, so it needs to be 
emphasized here that visual exploration is itself a form of inquiry as well as demonstration. See for example 
(Wolfram, 2002) for an example of exhaustive visualization as demonstration. The strengths of the method 
include the fact that humans have highly evolved visual sense, which facilitates insights from multiple 
representations and expands understanding of relationships. Thus, WEKA for example, displays 
visualizations early in the process of data exploration rather than as the last step after analysis. The main 
weakness is that visualization alone is not enough specific information to convince one of a relationship; so 
multiple methods need to be combined to tell a complete story of the data. 

These strengths and weaknesses are related to explanatory constructs suggested by the VPA data and 
shared here to stimulate thinking and discussion. In both the frog and bee assessments, a count of unique 
student ID’s on a table of prediction vs claims produced a basis for what might be called the ecological 
rationality (Gigerenzer, Todd, & ABC, 1999) of the performance space, a foundation for computing the joint 
probability of variables, for example as the a priori probabilities in a Bayesian analysis. Empirical 
probabilities computed from the counts provided a basis for making inferences about the cognitive states of 
the population viz the affordances of the space and in relationship to the world outside of the performance 
space. Eor example, a count of the change from prediction to claim options documented a shift in opinion, 
implying that the structure of the choices as well as the associated action patterns of the populations making 
those choices provided evidence of the thought processes that accompanied those decisions. The goal of 
analysis is then to reconstruct the most likely explanations of action patterns given the ecological rationality 
of the population. 

A second observation is that saliency of a particular action is not a property of the action alone, but has to 
be paired with an object, the action-object pair, as well as a context of the action, which leads to the idea of 
larger action phrases or motifs. Eor example, “opening” any page is an action, but “opening the pesticides 
page” is a specific action-object pair with more meaning. Eurthermore opening that page near the end of the 
assessment when making the decision about which claim to assert, further contextualizes the meaning of that 
action. The evidence from the two virtual performance assessment analyses suggests the possibility that the 
larger the unit of appraisal or motif, the easier it is to discover the variations in the action patterns of users; 
and the more the context is understood, the more that saliency can be associated with some particular 
intention or goal. 


5. PSYCHOMETRIC IMPLICATIONS FROM THE ANALYSES 

Psychometrics involves two major tasks: the construction of instruments and procedures for measurement 
and the development and refinement of theoretical approaches to measurement. With the advent of game and 
simulation-based applications for learning, the instruments and procedures for measuring learning and 
performance are shifting away from point-in-time (e.g. means taken on a slice of time) to patterns-over-time 
methods (e. g. trajectories evolving during some period of time). This moves the discussion around 
assessment from numbers to the structure of reasoning (Mislevy et al, 2012). The study presented here 
illustrates some of the implications of the new methods for theory and procedures needed to imply and 
estimate what someone knows and can do from game-based actions. The implications fall into several types: 
nonlinear relationships, rule networks, Bayesian probabilities, and semantic structure of actions. 

With log data from an educational game or simulation, complex nonlinear relationships can be discovered 
via symbolic regression and mathematically expressed with a good degree of precision. Eor example, in both 
the frog and bee assessments, a relationship was discovered between duration and score and could be 
expressed with precision. 

Differing performance strategies used by subgroups have a discoverable well-defined meaning and 
expression in terms of association rules and network structure that can be validated with performance 
outcomes and scoring. The relationships are complex, overlapping and nonlinear. However, if there is no 
constraint on utilizing resources in the virtual performance assessment space, then there will be considerable 
overlap of patterns by all users (everyone uses everything), making the discernment of action patterns more 
difficult. 

Rule networks can be discovered that are useful for making automatic inferences within the constraints of 
the rule’s confidence level. The rules in the VPA for example could classify that the student belongs in or is 
excluded from a particular performance group, or if the student was already known to be in a performance 
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group, then when time or action sequences are added to the analysis, scores can be inferred. Tuning up rule 
mappings requires people who are knowledgeable of the performance space affordances to make adjustments 
for causal and concurrent influences. Once tuned up, the rule network can help define a multileveled 
perceptron that can automatically categorize inputs within the constraints of cross-validation results. 

Prior probabilities for Bayesian scoring and other automated analyses can be based on the prior 
performances of cross-validation groups. In the VP As for example, predictions and claims data from the 
tested population provided a number of prior probabilities. 

Patterns of action-object use (i.e. semantic structure) have predictive value, and we suspect that larger and 
larger phrases and sequences of action-objects will increase their value by enlarging the salience of the 
actions. In the VPAs the digraphs and association rules for action-objects provide details for differences in 
strategy patterns, subgroup membership, and performance level. For example, the pesticides group spent 
more time than others inspecting the red frog, inspecting the green water, and discarding green water. The 
pollution group talked to the red farmer more than others. The parasites group talked to more to the scientist 
and the farmer from the yellow farm than others. These sorts of differences in action-object sequences can be 
used to classify a user during an assessment. 


6. IMPLICATIONS FOR DESIGN OF VIRTUAL PERFORMANCE 
ASSESSMENTS 

The following suggestions based on the analysis and findings reported here are intended to heighten the 
potential variation among test takers of a virtual performance assessment so that differences in strategies and 
resulting actions will stand out during analyses. 

Designers should plan for larger units of appraisal than the single record event with a time-stemp. If 
possible, build these recognized units into the application’s data collection mechanisms as second order 
appraisals. The automated recognitions can contain some noise and can also be constrained by windows of 
time within which all the constiuent action-objects must appear, including “the in-sequence appearance” of 
action-objects when necessary. Related to this observation, there needs to be a method for identifying action 
sequences that are unique to specific searches and solutions in the virtual performance space. 

Time measures should, in addition to time-stamping events, document the event duration (e.g. the start, 
duration, and ending as a unit) of salient action sequences and use a non-cyclical amount of time, to avoid 
introducing cyclical artifacts caused by the modulo mathematics of clocks. 

In visualizations, utilize an assessment frame-based reason to place sections of action-object pairs closer 
to each other. For example, if the variables were organized into action-object groupings related to the 
conceptual structure of the assessment, then time-based visualizations would reveal new patterns and 
insights. 

To avoid the problem of everyone displaying a similar “use everything” strategy, designers should 
consider utilizing “anti-scoring” penalties that would further restrict the range of scores to better align with 
highly valued action-object sequences OR have clearly defined outcome subscales that align with scores. 
Resource utilization behavior would change if there was a “limited resources” cost to using time or choices, 
which would lead to more differentiation in the action patterns. 


7. CONCLUSION 

Highly interactive, high-resolution log file data from virtual performance assessments show promise for 
documenting in new ways what students know and can do. Data mining, machine learning and symbolic 
regression techniques are effective tools for analyzing and making sense from the time -based records and for 
relating those to both automated and human scoring artifacts. New psychometric challenges are emerging due 
to the dynamics, layered resolution levels, and complex patterning of actions with objects in virtual 
performance assessment spaces. Learning analytics analyses are helping uncover and articulate the 
relationship of time-event appraisals, visualization structures and resource utilization constraints on the 
psychometrics of virtual performance assessments. 
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ABSTRACT 

This paper describes an exploratory study observing the use of Minecraft (a popular sandbox style online video game 
environment) in a high school English literature classroom. We use Piaget and Inhelder’s (1969) constructivist theories 
about the formal operational stage of development to interpret the concepts of plot and characterization in three student 
produced films that were created within the game environment. We claim that Minecraft can be used as a tool to assist in 
the cognitive development of its players through the practice of constructed learning. Minecraft’ s popularity implies that 
this type of knowledge production is preferred by youth for informal learning, and has vast implications for educational 
curricula. 
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1. INTRODUCTION 

The relationship between online video games and learning is an incredibly synergized and fashionable topic 
in academia, K-12 education, and industry. Scholars hold that online video games provide myriad 
opportunities for players to learn via problem solving, persistence, innovation, and a dynamic web of choices 
that motivate millions of players (Squire, 2005; Shaffer, 2006; Gee, 2007). These opportunities are available 
both to consumers and producers of online video games. 

James Paul Gee (2007) compelled many to understand the benefits of online video gaming and its 
embodiment of deep and meaningful learning practices. Gee was certainly not the first to advocate for online 
video games as a form of learning (see Ito, 2008), but certainly his dissection of the challenges and 
opportunities inherent in online video games has drawn attention to ‘good video games’ (Gee, 2007, p. 12). 
Gee (2007) praises good video games (i.e., the most commercially successful video games) and their ability 
to provide players with a space for meaningful skill acquisition and collective intelligence gathering. 

While academic interest in online video games and learning is almost as old as online video games 
themselves (close to 40 years old), it is only recently that scholars have acknowledged that the medium 
implicates a different set of learning practices and outcomes. Mizuko Ito’s (2008) complex analysis of 
computer games demonstrates that traditional views about learning and curriculum can dampen the benefits 
of games, and take motivation completely out of the equation. When game designers try to embed games 
with instruction-centered K-12 curriculum, the motivational, collaborative, and engaging elements of the 
game are mostly lost on its audience. Ito explains that many online video games that are created with 
educational purposes are, “[F]ocused on curricular content, rather than innovative game play” (2008, p. 93). 
These games emphasize external rewards and reinforcement for very school-like tasks (p. 94). Gee’s (2007) 
work points out that good online video games are not designed with learning scenarios in mind, rather they 
are naturally embedded with learning contexts. 

Good (i.e., popular) online video games — which don’t overlay traditional instructional practices onto 
gaming landscapes — present players with learning scenarios startlingly different from the structural elements 
of the typical K-12 learning environment. This study demonstrates that the knowledge produced in a very 
popular online video game {Minecraft) presents a radically different notion of learning — albeit one that may 
support cognitive development. 
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The success of online video games will depend on an acceptance of a new style of teaching and 
learning — one that is centered on exploration, production skills, and self (or collaboratively) authored 
artifacts (McGonigal, 2008; Gee & Hayes, 2011; 2012). 

We see the popular sandbox game, Minecraft as providing these opportunities. Like the “good” online 
video games we have described, collaboration and collective knowledge production are a large part of 
Minecraffs popularity. However, Minecraft is a unique game that promotes a very different type of learning 
scenario — especially in comparison to the games that others have focused on (Gee, 2007; Ito, 2008; 
McGonigal, 2008). In the following section, we will describe our motivation for choosing to study Minecraft 
in the context of the traditional high school classroom. 


2. MINECRAFT AND PIAGET 

Garnering millions in revenue (Duncan, 201 1), Minecraffs success is not evident from its digital graphics or 
its game narrative. Markus “Notch” Persson, the game designer, created Minecraft to be intentionally simple 
and open so users could interact with environments normally impenetrable in most other online video games. 
Duncan’s words best explain the draw of the game: “What makes Minecraft ‘work’ is a fascinating mix of 
the game’s aesthetic sensibility, its mechanics, its development history, and the creative activities of its 
players” (Duncan, 2011, p. 2). Unlike more structured game worlds, such as World of Warcraft, Minecraft 
presents players with an environment where successes are based on their creative and collaborative efforts. 

Minecraft shares characteristics with sandbox game worlds (such as The Sims) that are driven by the 
creative efforts of its players, rather than games that encourage a more structured narrative and set of 
competencies (e.g., first-person shooter games like Call of Duty). The game is also a descendent of digital 
authoring tools such as Seymour Papert’s LOGO that intended to encourage authoring by teaching 
programming to students. Whether gaming or programming (or both), these tools encourage a constructivist 
notion of learning, where players (or students) are given tools for learning via experimentation and 
“tinkering” rather than conveying specific content (Ito, 2008, p. 101). 

Others hold that Minecraft presents a dynamic space for learning via social constructivism, where 
collaborators demonstrate specific skills, but also give players the ability to “learn how to learn” (Banks & 
Potts, 2010, p. 6). The Minecraft environment encourages interaction with the system in both graphical and 
technical forms, and the community of players use these elements to create vast modifications and new layers 
to the game. As a learning environment, playing the game allows teachers to give students opportunities to 
show how creative they can be, while also working collaboratively with others in their classes. 

Although the modifications evidence the highly collaborative aspects of the game and the game culture, 
our focus is on the highly constructivist nature of the game itself. This aspect of the game is not the first of its 
kind — it draws on many predecessors and developmental traditions. Of the types of games she observes in 
her ethnographic work, Ito (2008) forecasts that construction games (that support the more constructivist 
style of learning and cognitive development) will most closely align with the economic and cultural needs of 
future learners, and has the potential to transform the traditional modes of K-12 learning. In her words: “If I 
were to place my bet on a genre of gaming that has the potential to transform the systemic conditions of 
childhood learning, I would pick the construction genre. With the spread of the Internet and low-cost digital 
authoring tools, kids have a broader social and technological palette through with to engage in self-authoring 
and digital media production (Ito, 2008, p. 115).” Here, Ito sees the construction games as a space for 
experimentation for the types of meaningful practices players need both in the classroom and the world. Like 
Ito, we agree that construction games give players the space to tinker with these concepts. Her mention of the 
“systemic conditions of childhood learning” also implies the need for a shift from the heavy instruction-based 
practices that dominate childhood. 

2.1 Constructivism in the Context oi Minecraft 

We see Minecraft as a sandbox game that is heavily entrenched in the constructivist model of learning. In the 
context of education, constructivism is a way of building knowledge through meaningful interaction and 
experimentation with content. This theory suggests knowledge and truth are not universal, but each 
individual ‘constructs’ knowledge through direct interaction with the content. 
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Although there are many scholars who are associated with this educational model, we focus on Piaget’s 
constructivist theory of cognitive development in this essay because of its lineage to digital learning (we 
explain below). We chose Piaget’s notion of the formal operational phase because there is little written on 
about the ways that technology might support transitions into this particular phase of higher order thinking. 
We also believed that Piaget and Inhelder’s (1969) writings on the “experimental spirit” — which is crucial 
element to the constructivist model of knowledge production — are most clearly aligned with the culture of 
Mine craft. We did see evidence of Piaget and Inhelder’s (1969) concepts in our data, but we propose in 
future analysis to observe these crucial social aspects. 

We use Piaget and Inhelder’s (1969) developmental theories about the formal operational stage to guide 
our analysis of the products of their game play. We argue that Minecraft can be seen as a tool for the 
exploration of concepts that aid cognitive development — whether the understanding of logico-mathematical 
relationships in the concrete operational stages — or in this case, the abstract understanding of hypothetical 
outcomes in the formal operational stages. Specifically, our data suggest that the students used Minecraft as a 
tool to explore the abstract concepts of plot and characterization that were not tethered to a specific work of 
literature, rather they demonstrated a more complex (beyond concrete) understanding of these concepts by 
considering alternative combinations of elements (while following general rules). 

2.2 Theoretical Framework 

Jean Piaget’s contribution to the understanding of cognitive development is vast and has exerted influence on 
a multitude of classroom and family practices. Although there is a multitude of work that detail his findings, 
our theoretical analysis is largely derived from his 1969 work with Barbel Inhelder, The Psychology of the 
Child. In this work, Piaget and Inhelder spend a great deal of the text reinforcing the central tenets of their 
theories: that children progress through cognitive schemata, both by their biological readiness (represented by 
an age range), and their assimilation of new information that assists in their progression through cognitive 
development. 

In the Piagetian notion, children must be given the tools and space to play with “reality” in order to 
challenge their current understandings of the logico-mathematical realities of space and time, and 
subsequently integrate this new knowledge into a more complex schema. A large portion of the text is 
dedicated to the earlier stages of sensorimotor (infantile) and preoperational (anywhere from a toddler to 
about seven years old) of cognitive development. Little is discussed (both in this text and in other contexts) 
about the formal operational stage, the “last” stage of development, where preadolescents progress to more 
abstract understanding of concepts, and can generalize to understand theories of behavior. Piaget and 
Inhelder describe this phase as, “[a] final fundamental decentering, which occurs at the end of childhood, 
prepares for adolescence, whose principal characteristic is a similar liberation from the concrete in favor of 
interest oriented toward the non-present and the future” (Piaget and Inhelder, 1969, p. 134). 

The authors describe this phase as the emancipation from content, where the young individual is able to 
integrate his or her knowledge of concrete operations, but in a much richer, dynamic and imaginative way. 
He or she is no longer constrained by ego-centrism, or tethered to the physical object being in front of them. 
They elucidate the difference between preadolescent and adolescent understanding of the behavior of space 
through a series of (confusing) experiments with both age groups. In their experiments, Piaget and Inhelder 
asked children in the concrete operational stage and formal operational to understand what specific 
characteristics of a rod (made of wood, plexiglass, or metal) would make it bend when a weight was hung 
from it. The rods were also of varying length and width. When the children in the concrete stage would test 
the rods, they would attribute the bending of a rod to its length, even if when compared to another, a shorter 
rod of the same material would bend just as easily. The concrete group would quickly make determinations 
about what characteristics made a rod bendable based on one variable (e.g., length, width or material), and 
one possible combination of the two. The formal operational group possessed a dynamic “experimental 
spirit” (p. 136), whereby an individual in this stage, will test all possible combinations of rods to consider all 
the variables (length, width, material), and control for certain variables (e.g., do two plexiglass rods of 
different lengths but the same width also bend with the same amount of weight tied to them?). 
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Although confusing in their description, these observations demonstrate that individuals are assimilating 
their knowledge into a new schema known as the formal operational stage. In this stage, individuals can 
create theories and construct concepts that are removed to concrete props. Individuals in this group are able 
to hold those concrete experiences in their cognitive stores while also designing new possibilities that are not 
explicitly derived from the objects in front of them. 

In all phases of his developmental model, children and adolescents come to understanding the function of 
reality through experimentation. For Piaget and Inhelder, knowledge is derived from action upon objects 
(Piaget & Inhelder, 1969, p. 155). The knowledge gained from this tinkering is to extract their properties, and 
logico-mathematic knowledge of how the objects function in space. Piaget and Inhelder call this the 
“experimental spirit”, which is the strongest in the formal operational stage (p. 149). It is this “experimental 
spirit” which is the crux of the Piagetian concept of constructivist knowledge building — that knowledge is 
assimilated through careful observation and testing via informal experimentation. 

In this way, we see Minecraft is a tool that offers students (or players) the ability to garner knowledge 
through experimentation in the constructivist sense. As we have discussed, Piaget and Inhelder’ s theories of 
cognitive development are at the foundation of the constructivist view of cognitive development and 
learning. In our study, the students assimilate new knowledge in the formal operational stage via 
experimentation and expression in Minecraft. 


3. RESEARCH METHODS AND QUESTIONS 

In early 2012, the authors collaborated with a high school instructor in New England, as a result of many 
shared conversations about Minecraft and its potential for learning. The high-school instructor proposed to 
use the game to explore the concepts of characterization and plot with a small sample of ninth and tenth (n= 
20) grade students in his English literature course, and the university partners (i.e., the authors) would 
provide a communal server for all to use as a game space. The high school instructor was interested in 
finding new ways of achieving his curricular goals rather than having students individually write their own 
stories or having them read a work of literature. The researchers were interested in understand the 
pedagogical value of using Minecraft in as a classroom tool. 

According to the instructor’s informal survey of the class, only one student was initially playing the 
game; thus, 19 out of 20 students were unfamiliar with the game and how it was played. After introducing 
them to the game environment, he then introduced the assignment. The desired outcome of this assignment 
was to produce an online video of a narrative work. This narrative would be produced and presented in a 3D 
film inside the game space. These 3D films are also known as machinima. The machinima would be 
developed by each group to demonstrate their understanding of the literary concepts. 

The high school instructor gave the students two options to achieve the assignment— they could use 
Minecraft to create their online video using game play captured using free software called Bandicam 
(“Home”, n.d.), or they could create a live-action narrative film using a camcorder. Eour female students 
decided to not to use Minecraft, while the other four groups (one of which didn’t finish the final film) chose 
to develop their stories using Minecraft. We know little about why these females chose not to use the virtual 
space, but we hope to explore these challenges in future research projects. 

The instructor carved five class periods for the students to capture their narratives using either Minecraft 
or the video camera. Prior to the first of these five class periods, the students collaborated outside of class to 
start to develop their storyline. The instructor gave the students a prompt that they could use to assist their 
creativity. The assignment was as follows: Parents are out of town and kid is being pressured to host a party. 
He/she agrees and the party quickly gets out of hand. 

Then, during these class periods, the students were given access to laptop computers in the classroom to 
practice their stories, capture pieces in online video to review, and then time to revise their stories. The all 
female group was also given time to capture their storyline using the camcorder. When the Minecraft groups 
were capturing their online video within the game, we used participant-observation to understand how the 
teacher interacted with them to facilitate students’ use of the game environment. 
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After completing the data collection process, we examined our field notes from participant observation 
described above, as well as the students' film productions, or machinima. We used Piaget’s theories about the 
formal operational stage of cognitive development to drive an analysis of the students’ narrative films; but 
also, we allowed relevant themes emerge organically from the data. Our analysis was to understand the use of 
Minecraft as an instructional tool with high school students. We reconsidered our initial inquiry into the use 
of Minecraft: How do high school students in an English class use the Minecraft environment to address the 
concepts of characterization and plot? 

In their work on ethnography and virtual worlds, Boellstorf, Nardi, Pearce and Taylor (2012) impart the 
notion that the data analysis process in ethnography should be guided by critical discussions among the 
researchers. As students of learning theory and cognitive development, our discussions about the data began 
to touch upon the grand theories posited by Piaget and Inhelder. Boellstorf et al. (2012) state that relevant 
theories should be "responsive to the data and research interests" (p. 162). As researchers, our interest is to 
help scholars and practitioners see the value of technology in the context of pedagogy and cognitive 
development. 


4. RESULTS AND DISCUSSION 

Rather than code the transcripts of the films, but we relied on the plot and character development concepts in 
the film to indicate the students’ understanding of their work. This study was highly exploratory in its nature. 
Our paper suggests that tools like Minecraft promote an understanding of such concepts that incorporate a 
broader array of possibilities, while offering an understanding of the concepts through construction via the 
game mechanics. The older students in our study took a complete departure from the suggestion prompt, and 
developed their own story, “A Burning Passion”. The story features Joseph, a young man who had the 
unfortunate experience of watching his parents burn in a fire when he was very young — a fire that he 
mysteriously caused. The first scene features him weeping by his parents’ graves, and refusing to go and live 
with his uncle because if Joseph lives with him, his uncle will soon meet a similar fate. As Joseph warns his 
uncle, we see a lightning bolt ignite a fire in the distance. Here the students offer us their version of 
characterization by introducing a character with a tragic flaw — everyone he loves is doomed to burst into 
flames. Although their instructor could have easily had students read Tom Sawyer to identify a similar type of 
character and complexity, the interaction with Minecraft gives the students an opportunity to experiment with 
a range of characterizations, in order to develop formula that help them to understand how this variable 
functions in the context of a story’ s plot. 

In their story, “The Hole”, another group of ninth graders introduce us to another type of characterization 
via Roy, a delusional young man who is restrained by his family because he insists on digging holes. Roy 
digs holes because he is instructed to do so by a fun-loving bunny rabbit, that only he sees. Beneath his 
house, Roy constructed an entire world where he and his bunny friend can cause destruction. When Roy 
expresses his distaste for the rabbit’s incessant chattering (about non-sensical things), he barks at the rabbit: 
“Who are you, anyway?” The rabbit answers: “But Roy, I am you!” This type of character development 
demonstrates a dynamic understanding of character and the range of possibilities that can be considered when 
formulating characterization. Here, the students create Roy, who is odd and anti-social, but who has control 
in his purpose (to create underworlds), even if his mind deviates from reality. The students have developed a 
character with multiple personalities, using the space of the game to explore a complex range of character 
traits. We see this as evidencing a dynamic understanding of the abstract concept of characterization. 

Another group created a horrific tale called, “Flesh Eating Predator”, where three friends are trying to 
find a party, but they seem to show up to an empty house with a creepy host (who is potentially the predator). 
Two of the friends, Anna and Caroline, are concerned with the appropriate social behavior (wanting to party), 
but their other friend, Kelly, seems intent on saying socially inappropriate things about her dog and her mom. 
In this video, we see the students experimenting with social norms for their age group. The character of 
Kelly, who says socially inappropriate things, might represent their fears about being socially outcast. Once 
again, we see the students experimenting with different types of characterization, not linked to any concrete 
prop, but representing an amalgam of real life and mediated experiences. 
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In both of the stories “The Hole” and “A Burning Passion”, we witness students developing characters 
that who struggle with character flaws. These flaws offer the promise of doom, but overcoming the flaws 
offer the promise of stability. In the case of “A Burning Desire”, the plot centers on Joseph’s struggle to be 
intimate with others, because he fears that they will catch on fire (as all the people who Joseph tends to love 
do). The development of the plot in the students’ film gives them a chance to explore the concept of 
characterization from a multi-dimensional perspective, rather than the static identification of elements that 
are offered in traditional texts. We see this exploration through the lens’ of Piaget and Inhelders’ formal 
operational stage. Here, the game space offers the students the ability to play with various formulae related to 
the concept of character. The open nature of Minecraft gives the students (or players) the tools to construct 
characters and plots that, while archetypal in their nature, represent their own abstract understanding of 
characterization and character development, and are not tied to specific text. “A Burning Passion” 
demonstrates experimentation with a range of possibilities in terms of a story’s plot. 

In “A Burning Passion”, Joseph’s continued struggle with his flaw (auto-pyromania?) demonstrates that 
students have observed many types of characters throughout their cognitive development, both via traditional 
print texts, films, and television. Joseph’s story is reminiscent of a superhero that struggles with his powers, 
and is isolated from the thing he desires most: love and intimacy. In the story, Joseph finally opens his heart 
to another (Jessica) and has a child, only to watch them both be engulfed in flames as the denouement to the 
tale. We could certainly discuss the implications of these aesthetic choices in the context of their adolescence, 
but our analysis is more interested in the ways that Minecraft provided the students with an opportunity to 
develop a plot in a way that demonstrates their transition to the formal operational stage. “A Burning 
Passion” demonstrates experimentation with a range of possibilities in terms of a story’s plot. Like the 
children experimenting with different rods, weights, and materials in Piaget and Inhelder’s experiments, the 
students in our study are toying with the different variables and combination of elements in the concepts of 
characterization and plot. 

In “The Hole”, Roy, a delusional young man, also struggles with his character flaw, this time presented as 
a mental illness. Although Roy develops an alternative personality that compels him to dig holes, and build 
an underworld (where the walls are lined with explosives), when his sister comes to rescue him, she is able to 
coax him back from his violent ways. In the final climax, Roy battles his alternate ego and destroys him in 
order to join the ranks of his more “normal” family. Again, the students experiment with plot lines to 
understand how characters resolve or succumb to their flaws. These archetypal stories demonstrate that the 
students have long observed these concepts in popular media and instructional material. In this case, 
however, Minecraft is a vehicle where they can experiment with those concepts in more abstract ways, ways 
we believe mark the maturation to more formal operational thinking. 

“The Flesh-Eating Predator” narrative is less resolute. Kelly is murdered by the predator; and her friends 
meet a similar end. We felt that this data was not as rich in its display of characterization and plot, but we 
think it has less to do with the students’ understanding, and more to do with the technological scaffolding that 
is required to use Minecraft. Our analysis of Minecraft has mostly focused on the narratives that our 
participants produced within the game. However, underlying the more aesthetic elements were the technical 
production skills that were required to create such dynamic narratives. If the students did not master the 
technical skills of the game, either through the scaffolding of their instructor (or via collective networks on 
the Internet, such as YouTube), then their exhibition of the more abstract concepts were not as successful. 
For example “Flesh-Eating Predator” is not as strongly developed as the other films. Although there are 
underdeveloped characters and unclear plot lines, one of the major flaws of “Flesh Eating Predator” was the 
students’ technical skill within the game. They used the same avatar for both a victim and a murderer, so it is 
unclear which character is which. Although all the students were amateurs, the weaker skill set in this group 
demonstrates the definitive link between technical skill within the online video game and the concept 
development. 
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5. CONCLUSION 

Our exploratory analysis revealed some of the possibilities Minecraft offers to learners who are 
experimenting with abstract concepts. Students repeatedly demonstrated an understanding of the concepts of 
characterization and plot that were much more dynamic than a simple identification exercise presented 
through a static text (e.g., Tom Sawyer). 

Although our interest was to use Piaget and Inhelder’s notion of the formal operational phase of 
development to understand the type of learning possible in Minecraft, we believe these data evidence a 
growing trend in the 2P' century classroom. We feel our analysis shows that 2P‘ century technologies such 
as Minecraft, provide students with the opportunity to construct knowledge in meaningful ways and in 
situated environments that are highly impactful. In fact, many scholars, such as Gee and Hayes (2011; 2012) 
suggest that these preferred modes of learning are more salient and relevant for the current global 
marketplace. 

The goal of any exploratory study is to understand the need for deeper inquiry, and we believe our data 
have evidenced the potential for Minecraft to provide meaningful learning scenarios of which others have 
discussed (Squire, 2005; Shaffer, 2006; Gee, 2007). As we proceed, we are trying to balance the open-nature 
of game with the culture of standardized assessment that drives the current educational culture. We believe 
that a multi-modal approach is needed going forward, with quantitative analysis using structured task 
completion within the game, and qualitative interviews to understand the creative process within the game. 
Also, in future study, we will observe and evaluate the nature of the social constructivist learning that occurs 
in Minecraft. We didn’t aptly capture this part of the gaming experience, and we acknowledge that it holds 
vast implication for how the game is experienced. 

This paper reinforces the notion that there is a tension between the knowledge production that is 
characteristic of the game, and the instruction-based culture that dominates the contemporary classroom. As 
digital technologies shift the type of epistemology and modes of production that hold currency in the global 
market, it will be necessary to resolve these tensions in order to provide students with more valuable and 
meaningful skills in their professional lives. We see this small project as evidencing a much larger 
transformation at work. 
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ABSTRACT 

Science on a Sphere (SoS) is a compelling educational display installed at numerous museums and planetariums around 
the world; essentially the SoS display is a large spherical surface on which multicolor high-resolution depictions of (e.g.) 
planetary weather maps may be depicted. Eascinating as the SoS display is, however, it is in practice restricted to the use 
of museum professionals; students (and for that matter, older museum visitors) are unable to create their own displays for 
the surface. This paper describes a working software system. Math on a Sphere (MoS), that democratizes the SoS display 
by providing a simple programming interface to the public, over the World Wide Web. Briefly, our system allows anyone 
to write programs for spherical graphics patterns, and then to upload those programs at a planetarium or museum site and 
see the result on the giant sphere. This paper describes the implementation of the MoS system; sketches a sample project; 
and concludes with a more wide-ranging discussion of our user testing to date, as well as strategies for empowering 
children and students with greater control of public displays. 


KEYWORDS 
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1. INTRODUCTION 

When educational technologists refer to "display devices", there is usually a tacit assumption that they are 
talking about flat screens-perhaps on a desktop or laptop computer, or mobile phone. Not all educational 
displays, however, fit this description. This paper focuses on one such unorthodox example-a particularly 
remarkable one called Science on a Sphere (SoS), created by the United States National Oceanic and 
Atmospheric Administration (NOAA), and installed in over 80 museum and planetarium settings around the 
world. [NOAA 2013] The SoS display, shown in Figure 1, is a large (1.73m diameter) solid-white spherical 
surface accompanied by four synchronized projectors; typically, the system is used to display (e.g.) planetary 
weather maps, animations of continental drift, the surface of the Moon, and other "spherical" graphics. 



Eigure 1. The Science on a Sphere (SoS) display surface at the Eiske Planetarium in Boulder, Colorado. In this photo, the 
sphere is displaying a geometric pattern generated by the Math on a Sphere (MoS) system described in this paper. 
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The graphics projected on the SoS surface are multi-color, high-resolution, and smoothly animated-in 
short, stunning. Still, the surface is "closed to the public" in the sense that while museum audiences can 
watch and enjoy graphics on the sphere, they have no accessible medium with which to create their own 
patterns for display on the sphere. In the case of the SoS display, this inaccessibility to the public is 
particularly regrettable, since youngsters could gain a provocative introduction to non-Euclidean geometry 
through interactive programming on a spherical surface. By producing their own patterns on the SoS surface, 
children (and interested adults as well) could encounter notions such as geodesics, intrinsic curvature, and 
spherical coordinates in the course of compelling, personalized projects. 

This paper describes a working (and publically available) system, called Math on a Sphere (MoS) that 
allows users to create graphical patterns that may later be displayed on an actual SoS surface in a 
planetarium. By writing short programs in a relatively simple language (based loosely on the syntax of the 
early Logo language), people can create beautiful graphical patterns on their own personal computer screen; 
these patterns may then be retrieved and displayed on the giant public surface in a planetarium or museum. 
Currently, our project is conducted with the collaboration of the Fiske Planetarium in Boulder, Colorado; the 
NOAA offices in Boulder; and the Lawrence Hall of Science in Berkeley, California. Eventually we hope to 
solicit the cooperation of many more institutions so that larger numbers of planetarium visitors can be given 
(at least temporary) control of these remarkable spherical public display surfaces. 

The remainder of this paper is structured as follows: the following (second) section will provide a 
structural overview of our software system and an explanation of how it can be used to communicate with an 
installed SoS system. The third section begins with a brief description of our language (in its current, still 
evolving, implementation). The section then sketches a scenario for creating a spherical design; in the course 
of this scenario we will touch upon some of the interesting aspects of spherical (as opposed to planar) 
geometry that are highlighted by designing graphics for a sphere. The third section concludes with a 
summary of our experiences with users. The final (fourth section) mentions key related work, and in effect is 
a more wide-ranging discussion of one of the key motivating ideas behind this project: namely, that large- 
scale, unusual, and (often) public displays can be made accessible to children for education and play. It 
should be noted that, we have previously discussed the MoS system in [Eisenberg 2012, Hsi and Eisenberg 
2012]; those papers focused on spherical geometry and an evaluation of our first student workshop. This 
paper, by contrast, is the first full explanation of the system architecture; it is also the first venue for 
presenting recent student-created examples, and for a more wide-ranging discussion of central educational 
issues raised by the MoS system. 


2. THE MATH ON A SPHERE (MOS) SYSTEM 

It is probably most straightforward to describe the MoS system starting with what the user sees: namely, the 
MoS web interface. A screenshot of the web interface (in the midst of an ongoing programming project) can 
be seen in Figure 2. The major components of the interface are the three windows seen in the figure: at upper 
left, an editor allows the user to compose programs and define new procedures for creating graphical designs; 
at bottom left, a command interpreter allows the user to type in lines directly (e.g., procedure calls that draw 
patterns on the sphere); and at right, an interactive sphere view window allows the user to see a graphical 
rendering of the sphere display itself. The sphere in this window, parenthetically, may be "grabbed" and 
rotated by mouse button presses and movements, so that the user can view all portions of the surface. 

We will briefly discuss the end-user graphics language (the one shown in the editor window) in the 
following section. For the present, one additional element of Figure 2 is worth noting here: the button labeled 
"Connect" toward the bottom of the figure. When the MoS website is loaded onto a suitably prepared 
machine at a local planetarium, this button can be used to activate a connection between the MoS web client 
and the giant spherical display at the museum. (As noted, the project currently has the cooperation of three 
participating sites equipped with the SoS display; and as the MoS system progresses to completion we plan to 
solicit collaboration from many more sites.) 

The web interface of Figure 2, and the original SoS system (running the sphere in the planetarium) 
constitute the two major "end portions" of the system: the first, written in HTML5 and JavaScript, is our own 
creation, while the second is the creation of NOAA. Sitting between these two end portions is the final 
element of the Math on a Sphere system: the local server. 
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This element (our own creation, like the web interface) sits between the web client and the planetarium 
sphere, and communicates programs to the sphere. In order to do its job, the local server must be installed 
"on site", on the same device as the planetarium's SoS system itself. 



Window: 

Type commands below and click 'run commands'. 


.run commands ' 


Command Line Window: List of Commands 


WebLogo Command Interpreter O Math 
weblogO 


gUse lighting preset positi^ 


Sphere Window: 

Qick and drag to spin the sphere. 


stoocecnrrund Frame ttumber: 302 Frame rate: 8.33 fps Frame size: 1 Data rate: O.OKB/s 
show/hide 2d canvas 
Server status: 


Connect to server at: server/postreceive ( Connect) Ditconnect 


Eigure 2. The web interface to the MoS system. At top left, the editor window; at bottom left, the command interpreter; at 
right, the interactive sphere window. The sphere rendering here is interactive; the "ball" may be turned using a mouse to 
view a given design from any angle. (The system can be freely accessed through the wehsite www.mathsphere.org.) 

In effect, the job of the local server is to act as the "glue" between the end-user programs written in the 
language interface of Figure 2, and the SoS system that displays the results of the program commands on the 
sphere. The local server sends to the SoS a "movie" of constant frame-rate composed of a number of 
individual frames in a standard image format (JPG, PNG, etc.). Each frame of this movie consists of a view 
of the sphere rendered into a 2:1 aspect rectangular flat image in the ECE (Equatorial Cylindrical 
Equidistant) projection [SoS 2013]. With the current implementation based on initial Canvas rendering, we 
use images of dimensions 1024x512 pixels, which (while not optimal) is visually adequate for the SoS 
system. 

The strategy used by the local server to send this "movie" is to maintain a circular buffer of images (100- 
200 has proved a reasonable choice) as concrete files in a certain directory on the server machine. The SOS 
system treats these files as an animated movie to play even though their contents are being constantly 
rewritten dynamically (in circular order - the "local server" keeps in step with the SOS system and rewrites 
the image files on disk as they arrive from the client). 

The overall architecture of the MoS system is summarized in Eigure 3, which sketches (at right) the web 
client, (at left) the pre-existing SoS system and (toward the center) the local server and its functions. 
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Figure 3. An architectural sketch of the MoS system. When a user runs their program from the web page (at right), the 
movement and image calculations are performed on this (client) side. The images are then posted by the local server 
(center) to the installed SoS system (at left) for display. Sequential images are posted in a circular buffer to achieve 

animation. 


3. CREATING A SPHERICAL DESIGN USING MOS 

In this section, we illustrate the use of the MoS system through a scenario in which a user creates an 
icosahedral pattern in the language interface, and then projects that pattern onto the planetarium sphere. In 
the course of working through this scenario, we will touch upon some of the interesting ways in which 
spherical geometry differs from the planar variety. Since earlier papers have discussed spherical 
programming at greater length, this section will be kept brief; the intention here is merely to provide 
sufficient background to motivate the discussion of the following section. 



Figure 4. An icosahedral design, created by the MoS language system and displayed on the planetarium sphere. 

3.1 Turtle Commands on the Sphere 

The central elements of our language system are based on the "turtle commands" of the Logo language, in 
which a programmable pen (the "turtle") can be steered about a computer screen through commands such as 

forward and right. 
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An excellent introduction to the mathematics of turtle geometry can be found in the classic text by 
Abelson and diSessa [1980], while Paper! [1980] gives an eloquent philosophical introduction to the Logo 
language design. For our purposes, we can imagine a scenario with a student who wishes to create an 
icosahedral pattern on her local planetarium sphere, as shown in Figure 4. (Specifically, the design in Figure 
4 is the projection on the sphere of an inscribed regular icosahedron.) The student is going to create this 
design using forward and right commands, just like she would using standard Logo commands on the plane; 
except in the case of the MoS system, a forward command moves the turtle along a great circle path on the 
sphere (rather than in a straight planar line). 

The Figure 4 design consists of twenty equilateral spherical triangles, arranged about the spherical 
surface. One immediate point to note, however, is that on a sphere-unlike the plane-triangles do not have 
interior angles that sum to 180 degrees. In particular, for the spherical triangles in Figure 4, the reader might 
note that there are five (not six) congruent equilateral triangles arranged around each vertex of the design; 
thus, each interior angle of each triangle is 72 degrees, and each of the equilateral triangles in the figure has 
interior angles that total 3*72 = 216 degrees. (For more description of the surprising differences between 
spherical and planar geometry, see the aforementioned text [Abelson and diSessa, 1980] and the discussions 
in [Eisenberg 2010, 2012].) 

The upshot of these considerations is simply that we need to create an equilateral triangle on the sphere 
with interior angles of 72 degrees (and exterior angles of 108 degrees). Space considerations preclude a fuller 
discussion (and again, these issues are discussed in earlier papers), but as it happens the expression for 
creating an "icosahedral spherical triangle" (as seen in Figure 5a) is: 

repeat 3 {forward 63.5 right 108} 

The student can now extend her design by creating five icosahedral triangles arranged around a single 
point: 

repeat 5 (repeat 3 (forward 63.5 right 108} right 72} 

This will produce the set of five triangles shown in Figure 5b. 



Figure 5. Creating the icosahedral display. 5a (left): a single spherical triangle. 5b (second from left): five triangles 
surrounding a point. 5c creating still more triangles. 5d (right): the entire icosahedron displayed in Figure 4. 


Now, the student could, if she wished, create the full icosahedral pattern by carefully moving the turtle 
along already-created lines and repeating the command that generated Figure 5b. Continuing with this 
strategy (albeit carefully) could in fact produce the pattern in Figure 5c and, eventually, the entire icosahedral 
pattern shown in Figure 5d. Finally, the last step of the scenario is for the student to produce the pattern from 
5d on her local planetarium display; by visiting the planetarium (which by assumption is a cooperating 
institution that has installed the local server to send images to its own SoS system), the student can now 
produce the image shown in Figure 4 earlier. 


3.2 Creating a More Elaborate Design 

Having created a "standard" icosahedral design on the sphere, it is now straightforward to use the features of 
the MoS language to extend or elaborate that pattern. Figures 6a and 6b illustrate this notion: by adding 
additional lines to the basic triangles of the previous example, one can create a more decorative icosahedral 
design, and display that design on the NOAA sphere (as in Figure 6c). This example represents, of course, a 
tiny initial fraction of the types of explorations that can be conducted with the MoS system. The purpose of 
this scenario has been merely to suggest the sorts of projects that one can realistically undertake. 
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The larger point-to which we return in the final section of this paper-is that we have now made a 
gorgeous spherical public display available for experimental use by anyone-children included-with access to 
a Web browser (and a cooperating local institution). 



Figure 6. Left and center (6a and 6b), we take the basic triangle structure of Figure 5b and elaborate the design with 
additional lines and colors. Right (6c): a multi-colored icosahedral design (based on the design in 6b), rendered on the 

spherical planetarium display. 



Figure 7. Student-created designs. Upper row: designs created by (in order) a 10-year-old, 12-year-old, and two 1 1-year- 
olds at the Lawrence Hall, Berkeley. Bottom row: designs created by Boulder middle school students, and displayed at 

NOAA Labs in Boulder. 


3.3 Current State of the MoS System 

The current MoS system is publically available through the website www.mathsphere.org. This version (see 
also [Eisenberg 2012]) represents a substantially improved and extended successor to an early prototype 
system (described in [Eisenberg 2010]). In particular, all of the elements shown in Eigure 2-the language 
editor, interactive sphere view, and command interpreter-are new to this version, and the local server has 
been substantially redesigned. Currently, it should be noted that our MoS system is still a "work-in-progress", 
with many elements still under construction for future public releases. (There are also, not unexpectedly, 
occasional bugs in the current version still to be tracked and corrected.) 

Our initial pilot tests of the system were conducted in spring 2012 at the Lawrence Hall of Science in 
Berkeley, California (Eigure 7, top row). Two separate workshops were held for elementary- and middle- 
school-aged children in the San Erancisco area, focusing on both spherical geometry and programming with 
the MoS system (see also [Hsi and Eisenberg, 2012]). Earlier this year, additional pilot tests were conducted 
with eight middle school students in Boulder; the students had 3 one-hour-long lessons in using the system 
(spaced over three consecutive weeks), and then created designs that were displayed for them and their 
parents at the NOAA labs (Eigure 7, bottom row). 

Our experience to date indicates that students are indeed able to make use of the MoS system for creating 
and displaying attractive (if mathematically simple) spherical programs. 


222 








lADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2013) 


Our next steps will be to work toward longer-term interactions with students and to use MoS to introduce 
somewhat more advanced concepts in spherical geometry (conceivably for high school or even 
undergraduate-level students). 


4. TOWARD DEMOCRATIZED PUBLIC DISPLAYS 

Our motivation for creating the MoS system described in this paper is really twofold. On the one hand, we 
are interested in providing an aesthetically appealing medium for introducing non-Euclidean geometry for 
middle- and high-school students; the sample scenario of the previous section, brief as it necessarily was, 
indicates the sorts of mathematical issues highlighted by the system. A second reason for creating the system, 
however, is less specific to mathematics per se, and more aimed at expanding the landscape of display 
devices (and display settings) available to students. 

We can begin this reflection with the subject of "making public displays available to children". Earlier 
researchers such as Brignull and Rogers [2003] and Peltonen et al. [2008] have reported on pioneering work 
investigating users' activities with large-scale interactive public displays. Our SoS system has particular 
features that make it distinct from that earlier work: it highlights specific educational (in particular, 
mathematical) issues, it is replicated in many sites worldwide (suggesting that a user could in principle create 
a pattern to be displayed at various public sites over time, or simultaneously), and it offers a dimension of 
programming (as opposed to "direct interaction", as in Brignull and Rogers' "Opinionizer" or Peltonen et al.'s 
"CityWall" projects). 

The MoS strategy described here-the strategy for giving students control of public display devices-is 
really a flexible mixture of "remote" and "present" interaction. Conceivably, students can spend a significant 
amount of time learning spherical programming on their own, and then (only later) display their ideas 
publically; or alternatively, one could imagine scenarios in which students explore ideas immediately (as in 
our workshops), at the museum or planetarium space itself. One might, of course, repurpose this fundamental 
strategy for other public displays. Eor example, one might create a "public opinion" space, analogous to the 
"Speaker's Corner" at London's Hyde Park, through which students (e.g., for the purposes of debate or 
political participation) can project text, graphics, and video on high-resolution screens in large public spaces; 
or students might be given (temporary) control over a scoreboard display at a sporting event. Naturally, there 
is a sociological and political dimension to such a suggestion: students might elect to project (e.g.) 
controversial or even offensive content through such a system. The issues raised by a public display of this 
sort touch upon fundamental issues of privacy and free speech. Indeed, it should be noted that even in our 
own MoS system, the capability exists for a user to create (e.g.) turtle-drawn text messages to display on the 
giant sphere; and conceivably, a diligent user could employ the system to project words on the sphere that 
other museum-goers might not wish to read. Once programmable public displays are made available to 
children (and to the public), that decision is attended by all the benefits and risks of free and open speech in 
other settings as well. 

Of course, there are many more examples of this sort of idea that might be tried, not all of them as 
politically sensitive. Eor instance, one might return to a planetarium setting, and create an interface through 
which visiting school groups can project patterns on the interior of a planetarium dome. Art museums might 
include specialized surfaces of various sorts, perhaps using geometric forms other than the SoS sphere; for 
instance, one might create an "SoS-like" cube, cylinder, or cone-shaped display surface (just to name a few 
examples), and allow visitors to create designs for those surfaces through their own "MoS -like" web 
interface. Such scenarios highlight once more the specifically mathematical aspects of the MoS project. 

There are many other technological challenges that could emerge from experiments along these lines. Eor 
example, the MoS system is based on a default scenario in which a single programmer creates a design to 
send to a particular sphere. We might imagine an alternative in which several programmers, working from 
several distinct web interfaces, each contributes a portion of a spherical design; for instance, one programmer 
might create a "background" design for the sphere, while others create programs for animated "foreground" 
figures that move over the background graphics. 

Or we might try implementing an MoS system in which users could write programs at the web interface 
and then broadcast those to one or more distant spherical surfaces; in this variant, it would no longer be 
necessary to be physically present at the local planetarium in order to run one's program on the giant sphere. 
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Conceivably, the user in this case might be able to watch the results of his running program via a webcam or 
similar remote viewing device.* Yet another possibility would be to employ lightweight handheld projectors 
(held by students themselves) that can be overlapped or superimposed to create large-scale displays or 
animations under distributed control from multiple users. Or-to imagine still another alternative-one might 
create an MoS-like interface to display graphics on a moving or dynamic surface (such as a hanging mobile). 

The intent of these examples is simply to illustrate the ways in which a strategy like that employed in the 
MoS system can be adapted to all sorts of other public and pervasive displays. The control of such displays 
can now be distributed; even a complex, large, or expensive display associated with a particular physical 
location can now be made available for web-based graphics and programming. We look forward to the day 
when museums and planetariums (among many other institutions) routinely install beautiful, specialized 
displays that have relatively "open access" for their visitors. 
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ABSTRACT 

The e-Textbook and e-Schoolbag initiatives have received wide attention and have been seen as the trend towards future 
education. This paper describes the framework of e-Textbook and e-Schoolbag including its conceptual model and 
system model. It further discusses the five interconnected categories that form the main parts of this Ecosystem construct. 
The five categories include the overall standards, e-textbook standards, learning terminal standards, learning tools 
standards, and learning services standards. 
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1. INTRODUCTION 

Digital learning tools have been improved with the development of information technology. E-Schoolbag and 
e-Textbook as digital learning tools have been implemented in education at a fast speed (Wu, Yang, & Ma, 
2013). Two recent issues of the Horizon Report (The New Media Consortium, 2010; 2011) pointed out that 
e-Book technology would mature in the next two to three years, and that it would have significant impact on 
the development of learning science technology in the next five years. According to the survey report from 
CLIFF Market Consultancy Ltd, over 50 countries plan to promote e-Textbooks & e-Schoolbags. As a result, 
great attention has been paid to the foundation of this technology, the development of standards of 
e-Textbooks and e-Schoolbags. Open E-Books Forum organization developed OEBPS Open E-Books 
Publishing structure standards (Open eBook Forum, 2011). International Digital Publishing organization 
developed ePub standard and launched ePub3.0 standard on Oct. 2011 (IPDF, 2011). In China, Founder 
developed CEBX standards. International Standardization Organization ISO/IEC JTCl SC36, in partnership 
with British JISC and China National Information Technology Standardization Committee have begun to 
research on standards for e-Textbooks, and research on e-Textbooks contents, framework and affordances. 

On the 22ed to 25th ISO/IEC JTCl SC36 plenary and working group meetings, Chinese representatives 
led the development of international standards of e-Textbook (Wu, Guo, & Zhu, 2011). Ministry of industry 
and information Technology of China (MIITC) and General Administration of Press and Publication of 
China have conducted research on e-Textbook standards, as well as established relevant research institutions 
to work on the subject. An electronic reading industry consortium was founded in Taiwan and an e-Book 
industry consortium was founded in Shanghai. Shanghai Municipal Commission of Economy and 
Information launched special research on e-Book industry. In Nov 2010, China e-Textbook and e-Schoolbag 
Standards Working Group (CETESBSWG) was jointly founded by the China National Information 
Technology Standardization Technical Committee of SAC (Standardization Administration of the People's 
Republic of China) and China e-Leaming Technology Standardization Committee (CELTSC) of MOE 
(Ministry of Education) to work on e-Textbook & e-Schoolbag Standards (ETESBS). Now, there are many 
e-Textbook and e-Schoolbag initiatives and pilots under way covering nine provinces in China. 
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In Europe, CEN Workshop on Learning Technologies launched the eTernity project to develop a common 
vision, frameworks and specifications for e-Textbooks for educational purposes to bring European 
stakeholders in education together(CEN WS-LT,2013 ). E-textbooks and e-Schoolbags have also been 
adopted by many schools in the US, South Korea, Japan, and Singapore. Standards research, industry 
development and educational application of e-Textbook & e-Schoolbag have also been launched. 


2. FOUNDATIONAL WORKS FOR THE CONSTRUCT OF AN 
ECOSYSTEM OE THE E-TEXTBOOK AND E-SCHOOLBAG 

Chesser (2011) traced the evolution of the e-Textbook and identified several emerging best practices in the 
technical and business aspects of the new class of products. An e-Textbook and iPad were used on a Pilot 
Program (Roberta & Sloan, 2013). Wu, Yang and Xiong (2013) wrote about the characteristics and 
functionalities of e-Textbooks, and Xu, Su, Wu and Wang (2013) conducted some initial research on the 
behavior patterns of classroom teaching with e-Schoolbags. However, there has been limited research on the 
standards, industry development, and educational applications of e-Textbooks and e-Schoolbags. 

It is clear that standards of e-Textbook and e-Schoolbag must be developed before the industry 
development and educational applications of the e-Textbooks and e-Schoolbags. Currently, the IMS learning 
resources standards, SCORM of ADL, CEBX of Pounder, and ePub of IDPP form a profile of reference 
standards. In 2010, Chinese industry standard for the format of digital publication CEBX was developed by 
Pounder. In terms of the system of standards, technology standards of China ICT in education have CELTS 
system structure (CELTSC, 2013). Pramework of the CELTS and Reference Model suggest the framework 
structure specification of online education (Zhu, Zhang, & Wu, 2009). Wu (2011) summarized the overall 
standards for e-Textbook and e-Schoolbag in China. Afterwards, the framework of standards for ITLET of 
SC36 was put forward (Wu, Guo, & Zhu, 2011). Wu, Yang & Ma (2013) then further elaborated three 
aspects of e-Textbook including the standards research, industry development and educational applications of 
the e-Textbook. Several steps were taken before the Ecosystem of e-Textbook and e-Schoolbags was 
constructed. Pirst, the demand for the applications of e-Textbook and e-Schoolbag in China were 
investigated. Workshops were organized in order to understand the needs of relevant industries of 
e-Textbook and e-Schoolbag. Needs assessments were conducted among a wide range of e-Textbook and 
e-Schoolbag stakeholders, including users, governments, domestic and foreign companies, international 
organizations, and research institutions. Application cases and field research were conducted to better 
understand the users’ needs for terminal equipment. These needs assessments helped produce project 
proposals, application case reports, and technology reports to construct the Ecosystem of e-Textbooks and 
e-Schoolbags. 


3. THE CONSTRUCT OE THE ECOSYSTEM OE THE E-TEXTBOOK 
AND E-SCHOOLBAG IN CHINA 

The construct of the Ecosystem of the e-Textbook and e-Schoolbag includes two parts, that is, the conceptual 
framework of the e-Textbook and the model of the e-Textbook and e-Schoolbag system. 

3.1 The Conceptual Eramework of E-Textbook 

E-Textbook refers to the digital instructional materials (e.g. textbooks, teaching reference books) for teaching 
and learning (Wu, Zhu & He, 2011). It provides tools to help users to read, teach and learn. E-Schoolbag 
refers to the personalized learning environment with e-Textbook readers, virtual learning tools and ubiquitous 
learning services (Zhu &Yu, 2011; Wu, Zhu, & He, 2011). The standards for e-Textbook and e-Schoolbag 
are used to solve the interoperability among learning contents, learning platforms, learning tools, and 
learning terminal (e.g., Human-Computer Interaction). Pigure 1 shows the conceptual framework of 
e-Textbook and e-Schoolbag from the perspectives of interoperability. 
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Figure 1. The Figure of Conceptual Model for e-Textbook and e-Schoolbag 


As shown in Figure 1, the interoperability of learning contents connects between learners and contents as 
well as between contents and contents. The interoperability of learning terminal refers to interoperability of 
hardware equipment, such as the specification of hardware and software interface, compatible specification 
and environment adaptable specification, especially human-computer interface. The interoperability of 
learning tools refers to the interoperability of virtual learning tools, for example, interaction and collaboration 
based on different virtual learning tools. The interoperability of learning platforms refers to interoperability 
of learning services, for example, how to share learning resources in different platforms. The standards are 
developed from five scopes, including e-Textbook, learning terminals, virtual learning tools, learning 
services and the general framework. 

3.2 The Model of E-Textbook and E-Schoolbag System 

E-Schoolbag system covers four parts: learning contents, learning terminals, learning tools and learning 
services, as shown in Figure 2. Among them, learning contents, including e-Textbooks, e-Books and other 
extensive information, are learning objects when learners use e-Schoolbags. E-Textbooks are the core 
contents. Learning terminals, supplying hardware equipment to support learning activities, are the main 
media when using e-Schoolbag to study. They carry learning tools and online learning services. Learning 
tools are provided with virtual learning tools. Learning platforms supply all kinds of remote services to 
e-Schoolbag system, such as resources push, activity managements and so on. Therefore, the framework 
model of e-Textbook and e-Schoolbag system can be divided into e-Textbooks, learning terminals, virtual 
learning tools and learning services. 
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Figure 2. The Framework Model of e-Textbook and e-Schoolbag System 

E-Textbooks. Learning contents mainly contain all kinds of e-Books for study, extensive information 
(pan resources) and e-Textbooks. E-Textbooks are the most important learning contents in e-Textbook and 
e-Schoolbag system. They are different from ordinary digital publications. They have features of readability 
and teaching and properties with relevance, rich media, interactivity and openness (Wu, Yang & Xiong, 
2013). To research e-Textbooks, we need to not only adopt merits of present formation of e-publications to 
settle down operational issues of e-Textbook contents, but also to introduce teaching traits to the aspects of 
definition, inner structural characterization and media presentation of e-Textbook metadata. 

Learning Terminals. Learning terminal of e-Schoolbag is the general term of e-educational equipment, 
which plays an auxiliary role in learning. With development of technology, this new kind of educational 
equipment enhances the function of e-Textbook, and supports different forms of learning and various 
subjects. Learning terminals are provided with a strong learning function. Some of them are equipped with 
open operating system, which supports the functions of capacity expansion and player. At present, there is a 
wide variety of learning terminal equipment, such as Pad, Table PC, Mobile Phone, PDA. The research on 
the digital learning terminals will avail to rule and guide the design, development and application of learning 
terminal. 

Virtual Learning Tools. The name “Virtual Learning Tools” came from learning tools used in the 
traditional education process. Virtual learning tools are educational software and platform for learning 
activities. They develop into two directions. One is to present traditional physical learning tools in digital and 
virtual formations, such as electronic dictionary and virtual ruler. The other is to optimize new tools to 
support learning activities, which are designed by combining the research results of cognitive science and 
learning theories, such as conceptual tools and groupware tools. In the aspect of teaching and learning, virtual 
learning tools can be divided to teaching tools and learning tools. 

Learning Services. The research on learning services of e-Textbook and e-Schoolbag system is mainly to 
solve the following issues: 

(1) To improve the interoperation between e-Textbook and e-Schoolbag system and other related 
support systems as well as to improve the flexibility, sharing and reusability of the system. 

(2) To solve interoperating problems between e-Textbooks and services related to e-Schoolbags, 
data exchange among different systems and technical obstacles between service suppliers and 
consumers of e-Textbooks and e-Schoolbags. 

(3) To improve flexibility of integration of system and to mingle the present learning technical 
system (ITLET system) through exchanging data with other systems. 
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4. E-TEXTBOOKS AND E-SCHOOLBAG STANDARDS AND PROFILE 

E-Textbooks and e-Schoolbag standard system consists of five categories of technical standards as the first 
phase of research and development of technology standards. They are E-Textbook and E-Schoolbag Overall 
Standards Cluster, E-Textbook Standards Cluster, Learning Terminal Standards Cluster, Virtual Learning 
Tool Standards Cluster, Learning Service Standards Cluster. 

E-Textbook and E-Schoolbag Overall Standards Cluster is about the total framework, the basic 
standards and specifications of e-Textbooks and e-Schoolbag. It includes the Definition of e-Textbooks and 
e-Schoolbag, System Eramework Model, the General Terminology, as well as a description of the reference 
relationship between standards. It provides a comprehensive guidance for the application of the standard 
system of e-Textbook and e-Schoolbag. 

E-Textbook Standards Cluster includes e-Textbook Information Model, Content Package, Content 
Aggregate, XML Binding, Metadata as well as Practice Guidelines. It has an extremely important role in the 
application and popularization of e-Textbook and e-Schoolbag. 

Learning Terminal Standards Cluster describes requirements of the electronic learning terminal part 
from the hardware, operating systems and software provisions. It consists of three main criteria, which are 
Electronic Learning Terminal Hardware Specification, e-Schoolbag Operating System Specification and e- 
Schoolbag Normal Software Specification. 

Virtual Learning Tools Standards Cluster specifies the requirements with virtual learning tools in 
system framework model of e-Textbooks and e-Schoolbag. It includes five main criteria, which are 
Classification Norms, Descriptive Specification, Assembly Specification, Aggregate Specification and 
Assessment Regulate of Virtual Learning Tools. 

Learning Service Standards Cluster specifies the concept, information model, interface and binding 
norms of learning service in the framework of the model. It mainly includes four standards. General 
Specifications, Information Model, XML Binding and Demonstration Guide. 

Currently, two initial specifications. Architecture of e-Textbook & e-Schoolbag and Information Model 
of e-Textbook, were approved as China national standards on the May, 2013. And Terminology of e- 
Textbook & e-Schoolbag, Profile of e-Textbook & e-Schoolbag related standards, e-Textbook Metadata, 
Specification of e-Schoolbag Terminal Hardware, Classification of Virtual Tools, Description of Virtual 
Tools and Overview Learning Service of e-Textbook & e-Schoolbag have their drafts. 


5. INTERFACE DEVELOPMENT, CURRENT APPLICATIONS AND 
RESEARCH OF THE E-TEXTBOOK & E-SCHOOLBAG ECOSYSTEM 
IN CHINA 

By far, CETESBSWG has had more than 50 institutional members taking part in the research on the 
standards for e-Textbook & e-Schoolbag. The members have developed products based on the standards 
related to e-Textbook, learning terminals, learning tools and learning services. Examples of some interface 
design are presented below. Ligure 3 and figure 4 are demos of the Interface of Chinese Language and 
Chinese History as e-Textbooks. 



Figure 3. Interface of Chinese Language. Figure 4. Interface of Chinese History 
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Figure 5 below presents another good example of e-Textbooks named iEnglish, which was co-developed 
by the Foreign Language Teaching and Research Press, ECNU, and Intel for K12 English classrooms. As can 
be seen the Eigure 5, a variety of tools including (1) notes, erasers, dictionaries, (2) the mark pen, (3) the 
label, and (4) the toolset have been incorporated to support students' learning activities. 



Figure 5. Toolsets for e-Textbook of iEnglish 

In addition, ECNU Press and HSJC have co-developed virtual learning tools, which have protractors, 
compasses and other tools belonging to mathematics. Eigure 6 shows the interface of virtual learning tools in 
math. 



Figure 6. Learning Tools from ECNU and HSJC 

As for learning terminals, CETESBSWG has the terminal technical specification draft for the technical 
requirements of e-Schoolbag Learning Terminals. And Intel, Dell, BenQ, Hanwang, Yifang and E-Ren E- 
Ben, which take part in the research on the standard, have their products of different PAD and Table PC as 
learning terminals of e-Schoolbag. These products obey the terminal technical specification draft, and OS has 
Androids, iOS and Windows. 

Last but not the least, e-Schoolbag software for classroom teaching has developed according to the actual 
needs of classroom teaching. The software consists of PC servers by teachers and teacher and students Pad 
terminal components. The main interface of teacher PC server-side, main interface of teacher’s and student’s 
terminal are shown as figure 7, figure 8 and figure 9. The main functions of teacher the PC side are monitor, 
teaching on classroom, quiz management, test analysis, the screen lock and screen broadcasting; the main 
features of teacher’s terminal includes lectures play, quiz management, quizzes, status monitoring, screen 
lock, etc.; the main functions of student’s terminal are handouts play, problem management, file upload, 
screen sharing, quizzes and so on. Some functions of student’s and teacher’s terminal come from learning 
service in the server of e-Schoolbag system. 
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Eigure 7. Interface of PC Server 


Figure 8. Interface of Teacher’s Pad Figure 9. Interface of Student’s Pad 


CETESBSWG with its partnerships has launched multiple projects related to the e-Textbook and 
e-Schoolbag Ecosystem project. Field research, structured questionnaires and individual interviews have 
been used to collect data to assess the effectiveness of e-Textbooks development and application process. 
Some initial findings indicated that teachers need to change their ideas of teaching, need to learn to use 
morphological characteristics of e-Textbook, and focus on the humanized design of e-Textbook (Wu, Lei, & 
Ma, 2013). 


6. CONCLUSION AND FUTURE 

This paper provides an overview of the Ecosystem of the e-Textbook and e-Schoolbag project in China, 
including its conceptual framework and system model, standards requirements, samples of interface designs, 
and applications. The construction of the standards of e-Textbook and e-Schoolbag will guide the 
development of e-Textbook and e-Schoolbag, and thus promote their applications in education. And 
standards research, industry development and educational application of e-Textbook & e-Schoolbag have 
gradually become an ecosystem of e-Textbook &e-Schoolbag in China. In order to effectively promote 
them, the International Open Forum on the Development of e-Textbook and e-Schoolbag Standards and 
Applications will open on Shanghai, China on 28th Nov. to 29th Nov. 2013. (Website is 
http://www.etesbs.org/). With the help of CETESBSWG, more standards of e-Textbook and e-Schoolbag will 
be developed, more products produced, and more projects for application carried out for the students and 
teachers in schools in the future. 
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ABSTRACT 

This study examined an instruction method for the improvement of information processing abilities in elementary school 
students. Current elementary students are required to develop information processing abilities to create new knowledge 
for this digital age. There is, however, a shortage of instruction strategies for these information processing abilities. 

This research proposes a method for teaching information processing abilities based on a problem-based learning model, 
and was tested with elementary students. The students developed an improved ability to create new knowledge and to 
present relationships with information through the process of problem solving. This study performed experimental 
research hy comparing pre- and post-tests with twenty-three fifth grade elementary students over the course of eight 
months. This study produced a remarkable improvement in information selection, information reliability, information 
classification, information analysis, information comparison, and information internalization. This study presents an 
improved methodology for the teaching of information processing abilities. 


KEYWORDS 

Problem Based Learning, Information Processing Ability, Blended Learning 


1. INTRODUCTION 

Because of the development of information engineering, the twenty-first century is being called the 
Knowledge-Based Society. The lifespan of information and technology is shortened, and it becomes difficult 
to use information that we learned in school; therefore, the ability to choose appropriate knowledge and re- 
compose it is needed in authentic problem situations (Evensen, Hmelo, 2000; Jonassen, 2000). Educators find 
new teaching methods by reflecting on existing school-based teaching methods to meet the demands of the 
times. One of these new teaching methods is the Problem Based Learning (PBL). 

To put it concretely, PBL uses authentic problems drawn from the learners’ life experiences. It increases 
the interrelationships of learning materials and allows students to develop a higher degree of thinking ability 
using in-depth interaction, concrete experience, and clue-compared traditional learning methods used in the 
course (Barrows, 1994). Learners are motivated toward the achievement of learning by the removal of 
humdrum memorization and are inspired to learn in a spontaneous manner (Aspy D. N., Aspy, C. B, and 
Quinby, 1993). In addition, PBL develops interpersonal and teamwork skills as it cements group members 
and invigorates interaction and teamwork in the process of problem solving. It is an improvement on existing 
traditional teaching methods. 

PBL has developed and is changing with development of ICT and PBL, web-based study (Richards, 
2001; Zumbach, Hillers, Reimann, 2004). PBL is student-centered. It emphasizes cooperative learning with 
other people and solves authentic, complex, and ill-structured problems in a web-based environment 
(Koschmann, Kelson, Feltovich, Barrows, 1996). PBL based on ‘blending’ makes it possible to search and 
share various pieces of information. It develops new forms as members of the community interact in various 
ways and learning atmospheres change. It uses e-mail, online boards, synchronous and asynchronous 
conferences. Learner-learner and learner-instructor interaction becomes active and extends the lesson beyond 
the classroom because cooperation and communication are needed to solve problems. 
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It becomes more effective because of the need to develop a process for information searching, for the 
analysis and solution of the problem and problem solution plan, for verification, arrangement, synthesis and 
presentation. 

There is an abundance of information on the Internet. Students can improve their creativity and problem 
solving abilities by selecting suitable material, analyzing, synthesizing, and remaking it into new information. 
We believe that learners expand their thoughts beyond limited learning by using ICT to transcend time and 
place (Korea Education Research Information Service, 2004). It is more useful to select knowledge and 
information by personal individuality or thought than to simply memorize what is provided to them. Students 
should have more than simply a quantity of knowledge; that is, ‘How much you know?’ Instead, information 
processing abilities - that is, ‘Can you make new knowledge using information?’ - are what we should be 
developing. The difficulty is that computer classes are taught as extracurricular activities, and ICT education 
isn’t taught in every school. In this situation, ICT education does not provide concrete examples and ways to 
improve ‘Information Processing Abilities’ because it is taught in literacy training classes. 

The purpose of this study is to improve information processing abilities through the solving of problems 
concerned with students’ situations using the computer. To improve information processing abilities, the PBL 
model is applied and combined tasks are presented. Students use computers to solve tasks and improve their 
information processing abilities in the process of PBL-concerned tasks. 


2. BACKGROUND THEORY 

2.1 Information Processing Abilities 

The Information Processing Abilities are based on the ‘Standard ICT skills of elementary-middle school 
students’ as defined by the Korea Education & Research Information Service (Korea Education & Research 
Information Service, 2004). 

Many suggestions are provided. Considering the test group, elementary school students, we define 
‘Information processes ability’ as the ability of the students to discover a problem about subject by 
themselves; and then research and study the problem as they solve it. In other words, the students set the 
learning objective, find information by themselves (with the assistance of the teacher), and solve the problem. 
All of this involves the ability to collect, create, remake information, and internalize it. 

2.2 Examination of the Literature 

Gallagher, Stepien & Rosenthal (1992) found that the problem-finding abilities of students who attended 
PBL increased in their study concerning the effects of PBL on problem solving. In addition, they indicated 
that structured problems are not needed in the process of problem solving and problem finding. They suggest 
that PBL is a way to solve ill-structured problems through compared consideration of structured problems 
and ill-structured problems. 

Sage (1996) said that PBL is an education approach for a constructed curriculum and classes on life 
problems. It is helpful to learners in that it improves critical thinking and cooperation in research and shows 
the characteristics and the effects of PBL on the learning activities of students as a development and teaching 
strategy. Most research applied ill-structured problems and target adult learners, so Sage (1996) wishes to 
discover if ill-structured problems are also suitable for young learners. He has been suggesting follow-up 
research about how the problems are developed. 

Achilles & Hoover (1996a) said that students do not have sufficient socialistic ability and time to solve 
problems through cooperative learning in a study conducted to discover the possibility of PBL as an 
education innovation in one high school and two middle schools. According to their study, PBL is not an 
innovation strategy for school education in general, but it is flexible and helps students respect each other and 
does improve their ability to think through cooperative learning. 

Williams (1992) criticizes PBL in a study at a medical college, reporting that, in general, it does not help 
students to develop strategies for problem solving. He therefore suggests an amendment for elementary and 
middle schools that provides students with the opportunity to watch the ways in which experts practice in 
situations. 
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This would help the students to learn effective problem solving strategies more quickly and be more 
helpful to students than simply applying PBL without such expert and successful examples. He suggests that 
PBL should be verified with respect to age, motivation level, and the level of achievement of the learners. 

Achilles & Hoover (1996b) suggest that the education standards of elementary and middle schools is 
improved as a result of applying PBL in elementary and middle schools. With PBL, the curriculum is 
integrated, alternative assessment methods are suggested, teaching methods are improved, and active and 
cooperative learning is enhanced. Their study, however, suggests that students have difficulties solving in 
groups when they practiced PBL; it suggests that students need training before PBL is applied. 

Richards (2001) insists, on the other hand, that PBL classes are very useful as an approach to the 
integration curriculum using the Web. In particular, he emphasizes that students can do reflection learning 
and learn overall aspects with respect to the leaning objective. Zumbach, etc. (2004) suggests that there are 
on-going attempts to integrate the Web and PBL, and to provide dPBL (distributed Problem-Based Learning) 
courses which integrate PBL, LBD (Learning by Design), and Web-assisted CSCL (Computer Supported 
Collaboration Learning) on the Web. 

The literature suggests that, with respect to learning, a new education paradigm is needed. Long-term and 
continuous PBL instruction is needed in order to develop positive effects and an improvement in information 
processing abilities. In the process of their study, learners can improve their information processing abilities 
and find relationships in information; they can structuralize and schematize it. 

Thus, this study is concerned with real life problems of elementary students and presents various 
discussion subjects. It focuses on information processing abilities that create new information and 
development the ability to express relationships in information using the computer. 


3. STUDY METHOD 

3.1 Subjects and Period of the Study 

This study performed with twenty-three fifth grade elementary students over the course of eight months from 
March 2012 to October 2012, and presented PBL problems eleven times. In addition, an ‘e-PBL board’ was 
created and used for this study. 

3.2 Assessment Method 

The prototype for an assessment method to measure ‘information processing abilities’ based on the ‘Standard 
ICT skills of elementary and middle school students’ was made by the Korea Education & Research 
Information Service (Korea Education & Research Information Service, 2004). Two computer education 
experts and one education evaluation expert examined the content validity to assure the validity of the 
assessment method and suggest any necessary modifications. Afterward, this study developed an 
‘Information processing ability assessment table’ and used it according to Table 2. 

3.3 Design of the Study 

This study focused on experimental research which compared the abilities of the students through pre- and 
post-tests. 

The subjects are the test group. This research analyzed the information processing abilities of students 
before they study. It also analyzed through pre- and post-tests after the students practiced tasks using the PBL 
process of seven steps based on the PBL model. This study process was: first, reports of learners before 
applying PBL are analyzed itemizing the information processing abilities and the actual condition as 
researched. Second, the class homepage for the Web-based PBL was opened and managed. Third, the 
students were given ICT literacy training on how to solve problems using the computer. Eourth, PBL was 
performed in order to improve the students’ information processing abilities. Eifth, after applying the PBL, 
the reports of the learners were assessed using the Information Processing Ability Assessment Table and 
changes in abilities from the pretest to the post-test were ascertained. 
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3.4 Apply PBL Process for Information Processing Abilities 

The PBL was based on the most well-known model of Barrows & Myers (1993). This study teaches seven 
steps for the PBL process in eleven tasks, and then analyses and observes the results to continually instruct 
insufficient parts of the Information Processing Abilities. 


Step 1: Provides a task related to life concerning time through website as shown in Table 1. There are 
eleven tasks over eight months. 


Table 1. PBL Task 


Task 

Task based on the Web 

Relation of life 

Taskl 

The causes and effects of yellow dust on 
daily life and a solution 

Korea issued a yellow dust watch because of the yellow dust 

task 2 

Ownership declaration of Dokdo by JapanJapan declared ownership of Dokdo. 

task 3 

Korea-USA FTA 

Signing of the Korea-USA FTA 

task 4 

The oil tanker called the Sea Prince sank 
off Yeo-su 

Visit affiliated sister- school in Yeo-su Kum-o island in June. 


Are there any ways to overcome the 


task 5 

destruction caused by typhoons and 
heavy rains every year? 

Destruction caused by typhoons in Korea. 

task 6 

What is the problem with North Korea 
having nuclear weapons? 

The nuclear weapons of North Korea issue. 

task 7 

What is the problem with the 
Screen Quarter System? 

The Screen Quarter System issue because it has disappeared. 

task 8 

Overcoming the IMF problem 

The IMF issue is taught in socials classes. 

task 9 

Netiquette 

Netiquette education is required because a problem of netiquette was 
happening on the school homepage. 

task 10 

Family trip plan to Geojedo 

There was a family trip planned to Geojedo around holiday 
on the 5th of the month. 

task 1 1 

Volcano 

The nature of volcanoes is taught in science class. 


Step 2: Each team creates a ‘plan for task performance’. 

Step 3: Team members divide the task into personal tasks based on the ‘plan for task performance’. 

Step 4: Students are taught how to collect the information that they need by themselves, how to write up 
sources to ensure reliability, and how to check for the information’s validity in solving the problem. 

Step 5: Students are presented with personal tasks in active time and complete the team task through 
discussion. Students learn to remake the information that they found through classification, analysis, and 
comparison in order to solve the problem. They then upload the result to the website and an analysis of their 
Information Processing Abilities is itemized. A steady improvement in the students Information Processing 
Abilities was observed. 

Step 6: Students listen to the various opinions of other teams at presentation time. 

Step 7: At the end of the team activity, the students make a reflection journal. It is used as a good way for 
the students to reflect on their work, to check what they have learned, and what they think about their study. 
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4. RESEARCH RESULTS 

This study applied PBL model to students in order to improve their Information Processing Abilities. It 
compared pre- and post-test to ascertain the difference of information processing abilities using an 
‘information processing abilities assessment table’. This study assigned tasks to students and recommended 
their using an e-PBL board for learning. It shows the efficacy of learning using an e-PBL board for research 
on the actual conditions and provides an academic atmosphere for this study. It shows the learning process of 
an authentically applied PBL model and verifies the effect on applying it by measurements before and 
afterward. Table 2 shows a comparison of the Information Processing Abilities of the students before and 
after applying PBL. 

Table 2. Comparison of Information Processing Abilities Based PBL Before and After. 


Items 

Assessment point of view 

Pre-tests results N=23, Post-test results N=22 
%=100 %=100 


Do students remake the collected content as they want 
and edit using word processors and so on? 


5 

4 

3 

2 

1 


5 

4 

3 

2 

1 


rT 

4 

8 

2 

0 

9 

N 

12 

6 

3 

1 

0 



17 

35 

9 

0 

39 

% 

55 

27 

14 

4 

0 

ICT 

Know- 

► Do students insert photos and pictures properly to help 

N 

2 

0 

0 

4 

17 

N 

17 

2 

0 

2 

1 

with understanding? 

% 

9 

0 

0 

17 

74 

% 

77 

9 

0 

9 

4 

ledge 

► Do students arrange the report using tables and charts 

N 

0 

0 

0 

0 

23 

N 

15 

3 

1 

2 

1 


that aid in easy understanding? 

% 

0 

0 

0 

0 

100% 

68 

14 

4 

9 

4 


► Do students upload document as attached files to the 

N 

4 

8 

0 

4 

7 

N 

20 

0 

2 

0 

0 


board of the class homepage? 

% 

17 

35 

0 

17 

31 

% 

91 

0 

9 

0 

0 


► Do students find information and material which they 

N 

2 

13 

2 

6 

0 

N 

9 

9 

3 

1 

0 


need using the Internet? [Information collection] 

% 

9 

56 

9 

26 

0 

% 

41 

41 

14 

4 

0 


► Do students not select unnecessary information while 

N 

3 

8 

4 

8 

0 

N 

9 

9 

4 

0 

0 


searching the Internet, but only what they need? 
[Information selection] 

% 

13 

35 

17 

35 

0 

% 

41 

41 

18 

0 

0 


► Is the selected content information appropriate 

N 

2 

13 

8 

0 

0 

N 

9 

9 

2 

2 

0 


for the task? 
[Information reliability] 

% 

9 

57 

35 

0 

0 

% 

41 

41 

9 

9 

0 


► Do students write sources for the information collected? 

N 

1 

2 

0 

4 

16 

N 

15 

1 

3 

2 

1 


[Information reliability] 

% 

5 

9 

0 

17 

69 

% 

68 

4 

14 

9 

4 


► Do students do information classification, comparison. 

N 

2 

7 

3 

6 

5 

N 

3 

12 

7 

0 

0 

Informationand analysis as they work? 

Processing [Information classification, comparison, analysis] 

% 

9 

30 

13 

26 

22 

% 

14 

55 

27 

4 

0 

Abilities ^ students collect valid information for problem 

N 

1 

13 

6 

3 

0 

N 

2 

14 

6 

0 

0 


solving? 

[Information validity] 

% 

5 

57 

26 

12 

0 

% 

9 

64 

23 

4 

0 


► Do students synthesize and remake the information 

N 

2 

7 

2 

8 

4 

N 

6 

5 

8 

1 

0 


through classification, analysis, and comparison? 
[Information synthesis and remake] 

% 

9 

30 

9 

35 

17 

% 

27 

23 

36 

4 

0 


► Do students present their thoughts based on the collected N 

1 

1 

0 

3 

18 

N 

2 

12 

5 

2 

1 


information using valid statements? 
[Information internalization] 

% 

5 

5 

0 

13 

78 

% 

9 

55 

23 

9 

4 


► Do the results of the problem solving achieve the goal 

N 

1 

7 

5 

7 

3 

N 

8 

10 

4 

0 

0 


of authentic problem solving to some extent? 
[Information completion] 

% 

5 

30 

22 

30 

13 

% 

36 

45 

18 

0 

0 

Scale - 

5spoints:very good; 4 points: good; 3 points: normal; 2 points: needs instruction; Ipoint: needs much instruction 
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4.1 Research on the Actual Condition of Information Processing Ability before 
applying PEL 

This consisted of a profile as in Table 2 and an analysis of the actual condition of the Information Processing 
Abilities of the students. It itemized and analyzed the student subjects before applying the PBL. We, first, 
focused on their Information Processing Abilities before applying the PBL. 

• About 40 percent of students used content from websites without editing. We therefore needed to 
teach the students about the editing process using a word processor to produce the content that they 
needed. 

• Ninety-one percent of the students did not use materials such as photos, pictures, charts, graphs, 
and so on to help in understanding the content. 

• Thirty percent of the students could not upload their report to the board of class homepage. 

• In particular, 86 percent of the students did not indicate an interest in reliability; that is, what 
information was reliable or what was the basis of information that they collected. It was necessary 
to check for the reliability of the information and develop a proper attitude toward written sources 
considering the fact that there is an explosive increase in knowledge and a great deal of unreliable 
information. 

• Ninety percent of the students could arrange information downloaded from the Internet; however, 
they were unable to express their opinions using the information, or use it as clues. They therefore 
needed to be instructed with respect to how to express opinions with the information found through 
a search of the Internet and how to use the information as clues. 

• Sixty-five percent of the students knew how to select information from what they collected, but 35 
percent of the students needed instruction. 

• Forty-three percent of the students were not able to accomplish the PBL tasks because they were 
too unskilled to classify, compare, analyze, synthesize, or remake. There is an obvious need, 
therefore, for the students to have Information Processing Instruction with respect to classification, 
comparison, analysis, synthesis, and so on. 

The results indicate the following for consideration. 

• There are some students who cannot make tables and presentations to compare information through 
ICT abilities. In addition, some students cannot append files when they upload information to a site 
for sharing. ICT Literacy Training, therefore, should be taught to improve students’ Information 
Processing Abilities. 

• There are many students who simply copy the content of web page without any editing. It is 
necessary, therefore, to teach students the process of editing using a Word Processor. 

• Many students produce a problem-solving report in a form that is difficult to understand. They need 
to be taught how to use photos, pictures, and tables to aid in understanding. 

• Most of the submitted reports did not show the sources of the content which made them unreliable. 
Students need to be taught how to write content sources to ensure reliability and validity. 

Many students just copy and arrange the contents without classification, comparison, or analysis. They 
need to learn how to process information. Students need to be taught how to collect information, classify it, 
compare it, analyze it, and synthesize it. This is needed so that students can re-create information. 

4.2 Comparing Information Processing Abilities Before and After Applying 
PBL. 

We analyzed the pre- and post-tests after the students had practiced with eleven tasks using the PBL process 
of seven steps based on the PBL model. We compared changes in their abilities using an itemized list from 
the ‘Information Processing Abilities’ of Table 2. 
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• Forty percent of the students used content without any editing from search engine. Afterward, over 
95 percent of the students remade the information as they wished using a Word Processor and so 
on. 

• Over 90 percent of the students did not insert photos, pictures, tables, graphs, and so on which 
could have helped in the understanding of the content of the report. Afterward, over 86 percent of 
students inserted photos, pictures, tables, graphs, and so on to support their reports. 

• Thirty percent of students couldn’t upload attached file to the board of the classroom homepage. 
After applying PEL based on Web, all of them could do that. 

• Fifty-two percent of the students used information that had been classified, analyzed, and 
compared. Afterward, 96 percent of the students could use information with these methodologies in 
the post-test. 

• In pre-test, 14 percent of the students were interested in reliability; they checked to see if their 
information was reliable and what the source of information was. Afterward, 86 percent of the 
students provided the source of their information to check its reliability. 

• With respect to the internalization of the information, only 10 percent of the students used searched 
information as clues. Afterward, in post-test, 87 percent of the students could use information as 
clues to support their thoughts after their classification, analysis, comparison. 

• Forty-eight percent of students knew how to select information that they needed and 35 percent of 
the students used the information without any selection. Afterward, however, 82 percent of the 
students were able to select information properly. There were no students who simply provided 
information without any selection process. 

• At the beginning, 35 percent of the students succeeded in completely solving the problem through 
selection, classification, analysis, comparison, synthesis and re-production. Afterward, 81 percent 
of the students’ completed the task very well. 


5. CONCLUSION 

This research used a PEL model to improve the information processing abilities of a group of elementary 
students. 

Let us look at the results of the pre-test in an analysis of the information processing abilities of the 
students before applying PEL model. First, About 40 percent of the students used content from websites 
without editing. Second, 91 percent of the students did not use any materials such as photos, pictures, charts, 
graphs, and so on to help in the understanding of their reports. Third, 30 percent of the students could not 
upload their report as an attached file to the board of the class homepage. Fourth, over 75 percent of the 
students were accustomed to collecting information, but did not have sufficient ability to classify, compare, 
and analyze the collected information. Fifth, and not significantly, 86 percent of the students had no interest 
in the reliability of their sources or what was the basis of the information that they had collected. Sixth, 90 
percent of the students could arrange information from Internet, but they were unable to express their 
opinions using the information as support. Seventh, 65 percent of the students knew how to select 
information that they wanted to collect, but 35 percent of the students needed instruction. Eighth, 43 percent 
of students were not able to complete the PEL tasks because they were unable to classify, compare, analyze, 
synthesize, and remake. 

These are the results of this study. First, there were very significant improvements in the students’ 
abilities. The percentage of ‘information selection’ abilities inereased from 48 percent to 82 percent. 
‘Checking of information reliability’ increased from 14 percent to 86 percent. ‘Information classification, 
analysis, and comparison’ and ‘internalization of information’ abilities inereased from 52 percent to 96 
percent. The abilities involved in ‘information collecting’ and ‘checking for information validity’ increased 
only slightly from 88 percent to 96 percent, and from 74 percent to 96 percent, respectively; probably 
because the learners had these skills to some degree before the PEL model was applied. Second, the number 
of students who inserted photos, pictures, tables, and graphs in their reports to help in its understanding 
increased from 9 percent to 86 percent. Forty percent of students used content from websites without any 
editing; but, after the application of the PEL model, over 95 percent of the students remade the information 
as they wanted using a Word Processor of some kind. Third, their information processing abilities improved 
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because this was not cramming method of teaching; but, instead, the process of solving a problem and 
making a ‘task plan’ based PBL. It is a teaching-learning method that improves information processing 
abilities. 

There are some implications from this study. First, it is helpful to apply the PBL model to improve 
information processing abilities; therefore, program research and development to improve information 
processing abilities using various teaching-learning models (Goal Base Scenarios, Action Learning, etc.) 
based on learning by doing in addition to PBL should be continued. Second, ICT literacy training should 
precede PBL work if information processing abilities is to improve. The ‘2009 Revised Curriculum’ being 
applied in Korea at this time does not allow enough time for ICT literacy training. Each school, therefore, 
should provide ICT literacy training time as part of the national curriculum in order to improve the students’ 
information processing abilities and make them suitable for the information age. 
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ABSTRACT 

This paper focuses on the multiple agents of educational change associated with the implementation of ICTs in 
elementary schooling. The focus of the paper is on emergent patterns of change, i.e. the way technologies are adapted 
over time in different configurations that involve both pupils, teachers, activities and the different resources used in the 
classroom. The paper focuses on the concept of socio-material bricolage (Johri 2011) as an approach to understanding 
how digital devices contribute to constructing both relevant and innovative practices in teaching and learning in schools. 
The concept of bricolage is based on the idea that teachers and learners develop their practices through the ad hoc use of 
available artifacts - i.e. through improvisations where digital devices participate in shifting relationships with available 
resources and relevant actors. The paper is a theoretical exploration of the use of iPads amongst teachers and learners and 
builds on data from a school development project in Denmark where five classes of seventh graders were given iPads on 
a one pupil one device basis for the school year of 2012-13. 


KEYWORDS 

Mobile learning, emergent patterns of change, bricolage 


1. INTRODUCTION 

The integration of ICT in teaching and learning is usually intimately associated with the innovation and 
transformation of educational practices (McCormick & Scrimshaw 2001, Pelgrum & Law 2003). One of the 
questions arising from this association is how to identify the agents and practices of change - if we assume 
that ICTs are not isolated agents in shaping change and guaranteeing reform. 

Change is a broad concept that can be conceptualized both from the perspectives of policy, of teachers’ 
professional practice and through research in educational contexts (Ottesen 2006). Central to policy 
perspectives and to some aspects of research is the discourse of implementation which focuses on linear and 
efficiency changes to practice in which technology is a significant actor (Iliomaki 2008). Kearns (2002) for 
instance argues, on the basis of an international study of policies for the implementation of ICT in schools 
that many countries have moved through at least two phases in their attempts to integrate ICT, where the first 
phase involves the investment in hardware and the second phase a more strategic integration focusing 
increasingly on teachers and educational strategies. Similarly, Bauer and Kenton (2005) propose, on the basis 
of Hooper and Rieber (1999) that teachers’ journey towards the use of ICT can be identified as a five phase 
process involving familiarization, utilization, integration, reorientation, and evolution. According to Hooper 
and Rieber teachers often do not proceed through the utilization stage, and into the ‘breakthrough’ of ICT 
integration, due to for instance lack in engagement in the technology. Though the context of teachers’ use of 
ICT may have changed since the early 2000s, evolutionary and linear approaches to educational change thus 
still abound, though research has to some extent shifted its focus from ‘implementation’ to the significance of 
the social distribution and enactment of change (Windschitl and Sahl 2002, Burden et al 2012). 

Though it may be useful for research as well as for policy to conceptualise educational change through 
the idea of phases, implementation and evolutionary progress, these approaches to change often fail to 
explain exactly “how individuals make leaps of progress” (Windschitl and Sahl 2002) and how practices can 
be organized to support educational transformation. In addition to this, it is not always clear what exactly 
characterizes the full integration of ICT and how this should affect the strategies envisioned and implemented 
by schools and teachers. 
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Finally, the focus on change as an end product rather than a process and an emergent practice may 
overlook the daily shifts and reconceptualisations in educational practice that involve ICT as an actor in 
change. In effect, an evolutionary approach to change may therefore marginalize or fail to acknowledge the 
ways in which teachers and learners constitute change as an improvisational rather than a designed, linear 
and rational action. As a consequence, a perspective is needed that will capture the emergent development of 
practices as well as the ways in which practices are situated in specific school cultures and educational 
strategies. Such perspectives may be provided by ethnographical studies and studies in the materiality of 
learning that stress the improvisational rhythms of change in emergent practices of teaching and learning. 


2. TECHNOLOGIES AND THE SOCIO-MATERIALITY OF LEARNING 

Recently, a number of publications have underlined the significance of understanding educational practice 
and change from the perspective of ‘material practices’ and the ways in which educational spaces are 
constituted through social agency (Fenwick and Edwards 2010, 2012, Johri 2011, Sprensen 2009, McGregor 
2004, Lawn & Grosvenor 2001). A number of these publications take their inspiration from practice theory, 
including activity theory (Engestrom 1999) and Actor-Network Theory (ANT). The significance of these 
studies to the study of educational change lies in the focus on change as emergent practices in which social 
and material aspects of teaching and learning are intertwined. Inspired by Johri (2011) I use the term socio- 
material bricolage as an analytical framework to describe the entanglement of material and social aspects of 
teaching and learning with technologies which will underline the emergent and improvisational nature of 
change when seen from a practice perspective. 

In his discussion of the role of socio-material theory for learning Johri draws on a number of sources, but 
significantly Latour (2005) and Levi-Strauss (1967) to define how the concept of socio-material bricolage 
can be used as an analytical framework for understanding emergent practices with learning technologies. 
Johri argues that the study of learning technologies can be qualified by looking at practice through the lens of 
socio-materiality, which may contribute to moving the perspective of research away from technological and 
empirical determinism and into the ways in which social and material agency creates constitutive 
entanglements in learning practices. Johri uses the term socio-material bricolage to show how artefacts derive 
their meaning and are constituted through social agency, i.e. how tools become ‘tools in socio-material 
context’ or socio-material assemblages. Building on Levi-Strauss, Johri argues that educational actors often 
use the tools that are available to them, i.e. they make do with what is at hand, rather than sticking to planned 
approaches that would require them to use tools that are not immediately available in their local space of 
practice. In this sense assemblages - or bricolage - of tools in practice become emergent designs of 
technology in use, adapted over time. Johri proposes that the idea of socio-material bricolage can help us to 
make distinctions between practice-as-designed and practice-as-practiced, where the latter highlights the 
improvisational and emergent aspects of practice. The concept of bricolage may therefore support 
understandings of “the emergent and socially and materially intertwined nature of human practices” (2011, 
212). I shall argue that both teachers and learners in the local context where I did my fieldwork were 
continuously engaged in improvisational practices through bricolage, i.e. with combining different kinds of 
activities, technologies, and other classroom resources and that both teachers and learners enacted and 
understood change by engaging in these improvisations. In these improvisational rhythms the iPad became a 
central actor because of its flexibility and availability, i.e. the fact that it is at hand and immediately 
accessible to the user. In the following I shall describe and discuss these improvisational rhythms of change 
in which iPads are entangled in practice, and the ways in which these practices can be used to understand 
change as a shifting and emergent process rather than a linear and designed process of implementation. 


3. THE IPAD STUDY 

Middletown is a lower secondary school in the west of Denmark in a municipality that has a high profile in 
school development and integration of ICT into education. The school has recently been through a process of 
merge where pupils from an associated school for children with special needs were integrated into the school. 
The school has not had a prominent ICT profile before the project started, mostly due to budget restrictions. 
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The school teaches pupils at three levels, i.e. 7*, 8* and 9* year of schooling. Pupils come to the school 
from other schools in the area, and it is therefore important for the school to accommodate pupils from 
different neighborhoods and backgrounds. 

The project followed 5 classes of 7 graders (aged 13-14) who were given iPads on a one pupil one device 
basis to keep for the entire school year of 2012-13. Two of these classes were special needs classes and the 
research project focused on how the use of iPads in teaching and learning could support inclusive learning 
environments. My research focused mainly on pupils’ learning, but also included understanding the ways in 
which teachers reorganize and redefine their teaching as an aspect of having technology accessible on a daily 
basis in classrooms and at home. I followed pupils in all five classes for three months at the beginning of the 
school year observing them in their daily lives in school and interviewing groups of pupils. In addition to this 
I followed teachers in classes, at meetings and during breaks, lunch hours and introductory courses. I had 
numerous informal conversations with teachers and did formal interviews with the group of teachers who 
taught the seven graders as well as individual teachers responsible for the classes. 

Educational change was embedded in this project as a result of linking school development with new 
forms of device ownership and usage connected with the iPad. According to Burden et al. (2012) and 
Melhuish and Falloon (2010) affordances of the IPad include ubiquitous and easy access, portability, and 
personalized and individualized experiences. These affordances should not be understood as properties of the 
technology itself, but as possibilities for practice and agency in educational contexts, for instance through 
situated, just in time usages of the technology. I am proposing that the flexibility and accessibility of a 
mobile device such as the iPad enhances its potential for participating in improvisational teaching and 
learning rhythms, as it is both at hand and will easily adapt to different kinds of teacher and learner needs. 
This is also what qualifies its use in the construction of emergent inclusive learning environments. 


4. HOW TEACHERS CONEIGURATE CHANGE THROUGH 
IMPROVISATIONAL RHYTHMS 

In Middletown School teachers were given a head start in trying out iPads for teaching and learning in the 7* 
form classes. As courses were planned for teachers in June and it was felt that they needed the summer 
holiday for familiarizing themselves with the technology, they received the iPads two months before the 
pupils. However, the courses given to teachers were primarily traditional ‘instructivist’ courses that did not 
directly link tablet ‘affordances’ to teachers’ practices. After the courses teachers were expected to develop 
their teaching with the iPads on their own or through collegial collaboration in the time usually given for the 
preparation of lessons. 

As argued by Johri (2011) and Lawn and Grosvenor (2001) teachers tend to improvise when using 
resources in the classroom for teaching. Lawn and Grosvenor (2001) focus on the ways in which schools 
have become new spaces for consumption since the 70s and how this affects the presence and use of 
technologies and other kinds of artifacts in the classroom. Material cultures are thus abundant in the 
classroom where different kinds of tools have historically replaced each other - or where they coexist and 
interact as part of teachers’ use and production of relevant resources for teaching and learning. In their 
account of teachers’ use of these artifacts Lawn and Grosvenor argue, similarly to Johri (201 1), that teachers 
generally use the tools that are accessible to them, they ‘make do and mend’ and thereby craft their own 
assemblages of learning resources as a prerequisite for being able to operate in the classroom. This is not to 
say that teachers do not plan or reflect on their teaching, but to underline that teachers often craft their 
approaches to teaching and their use of resources from what is at hand and available to them on a daily basis 
in classrooms. Crafting and constructing relevant learning approaches as well as assemblages of materials 
that are subject and learner relevant are, I shall suggest, deep and useful skills that are extremely relevant in 
many teachers’ daily lives, and ways in which teachers can both locate themselves in a material culture of 
learning and in the history of material cultures in the classroom. These improvisational strategies may 
intensify when teachers are confronted with a new technology and do not feel comfortable with the 
functionalities of the technology at hand or its pedagogical uses. 
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In addition to identifying teachers’ craft approaches to the learning resources at hand, Lawn and 
Grosvenor describe how skills such as writing are enacted as systems of related technologies (2001, 125), i.e 
as assemblages of learning materials (for instance pen and paper or a pc and relevant software) that will 
support teachers and learners in operating a particular skill. Skills are in this sense both materially and 
historically enacted - and can be conceptualized through the concept of socio-material bricolage - I will 
argue. Thus, the learning of skills cannot be understood outside the constitutive entanglements of activities, 
actors and materialities that operate through both practice-as-designed and practice-as-practiced. 

In my fieldwork I observed how teachers acted in classrooms where the iPad was one of the resources 
present - as well as a resource that was given a relative priority over other kinds of resources in the 
classroom. I also listened to teachers’ reflections on the ways in which this technology had changed their 
daily lives with the learners, as well as during breaks and meetings and at home. One of the things that the 
teachers continually stressed was that the courses given in June had not really prepared them for the ways in 
which the tablets could enter into their teaching practices and their professional and personal lives. This was 
partly due to the fact that the courses had not actually focused on the needs, challenges and rhythms of their 
actual teaching, but had rather introduced them to the functionalities of the technology, including apps etc. In 
contrast to this, teachers had generally acquired their knowledge of how the iPads could be relevant for 
learning by having them at hand, i.e. by exploring their functionalities and uses in practice and by observing 
and reflecting on pupils’ uses of them in different learning contexts. These approaches to the iPad may in 
some ways be described as experiential and explorative, i.e. as being embedded in teachers’ rhythms of 
explorative uses, and their reflections and reconceptualisations of use (Burden et al. 2012). The idea that the 
change potential associated with the iPad becomes constituted mainly as a result of its being at hand, in the 
classroom, is significant in that it underlines the importance of the continuous presence of the technology in 
the spaces where teachers perform their professional roles and where pupils learn. In being at hand, the iPad 
thus became one of the resources that teachers could include in their craft approaches to teaching. 


5. HOW LEARNERS CONEIGURATE CHANGE THROUGH 
IMPROVISATIONAL RHYTHMS 

Compared to teachers learners are often less likely to improvise, given that teachers to some extent 
orchestrate and supervise their work in the classroom (and in other places, for instance in group work and 
with homework after school). However, to the extent that pupils are allowed to work in their own pace and 
with their own projects, questions and agendas, space may be allowed for increased improvisation and 
experimentation. In addition to this, the availability of resources in the classroom may, I shall suggest, 
inspire pupils to collect, combine and explore the resources that are relevant for their particular learning 
needs. This entails, however, that teachers are open to the diversity of approaches to content and learning that 
is embedded in this practice of bricolage. 

Though learners may not have equal opportunities for improvising in the classroom when compared to 
teachers, children may be as inclined to experiment and improvise as teachers - though possibly for different 
reasons. When using technology in their spare time, children, especially boys, are for instance often seen to 
experiment in order to understand the relevance of the technology for their activities and aims, and in order to 
play and interact (Hou et al. 2006). This is equivalent to the crafting and bricolage involved in teachers’ 
professional work as described by Johri (201 1) and Lawn and Grosvenor (2001) above, though children may 
improvise to learn, not to teach. As with teachers, pupils seem to make do with what is at hand, and to adapt 
different resources to their immediate needs and activities. This is the way that pupils both orchestrate change 
and maintain stability in their ways of learning. 

In my fieldwork I observed that pupils regularly engaged in bricolage when working on their own or 
together in the classroom. In these bricolages the iPad was often a significant and central element that 
supported them in exploring the multifunctionality and flexibility of the technology. In the improvisations 
enacted by pupils the iPad was one of the resources that was continuously at hand, and that pupils could 
adapt to their different strategies of assembling resources and connecting them through learning activities. 
Enacting bricolage was in fact one of the ways in which the iPad was implemented into pupils’ emergent and 
shifting processes of learning activity. 
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Examples from my data show that in a number of cases the activity of bricolage was used to enhance, 
stabilize and make learning processes more efficient and relevant for the individual pupil or the group when 
engaging in learning activities. In some cases the improvisations also turned into rhythms of change that 
extended the established routines of stabilizing, enhancing and making learning more efficient. Examples of 
these improvisational rhythms will be given below. 

5.1 Stabilisation, Efficiency, Enhancement 

One of the routines in pupils’ work in the classrooms in Middletown school is using materials for copying, 
translating, selecting and checking information. Books, for instance, often had a significant role in 
‘containing’ information, illustrations, tasks etc. that teachers would refer to when asking pupils to work on 
assignments and tasks. Eor pupils, books would therefore act as a point of reference for the task set by the 
teacher, which is why books would often be included in pupils’ bricolage activities. However, in pupils’ 
practices, internet sites such as Google and Wikipedia would often be alternate, significant points of 
reference for knowledge and information that could enhance and expand the knowledge found in books. 
Therefore books and iPads were often combined in pupils’ bricolage strategies in order to access different 
kinds of knowledge and to compare them. 

Working with information and knowledge through repetition, translation and visualization was one of the 
chain of processes that was often initiated by teachers in Middletown School. These processes would be 
enacted through activity based assemblages of materialities (books, iPads, jotters), and materialities would 
often be the end result of the tasks (for instance posters). Teachers would for instance ask pupils to make a 
poster about the solar system or a specific country, they would asks pupils to make a film about themselves 
or take pictures or films to make a story. 

Leander and Lovvorn (2006) describe how literacies are often enacted through circulations between texts 
when for instance pupils move (i.e. transfer and translate) texts from one, more authoritative, textual space - 
for instance the whiteboard or book - onto their own printed pages. These constitutive enactments of 
literacies contribute to producing learning rhythms and processes through pupils’ transference of knowledge 
or information from one material resource to the other. In my study I observed similar rhythms of 
transference and translation in the circulation of materialities in the classroom, where transference and 
translation were often enacted through the piling or linking of resources in assemblages (bricolage) that 
included iPads, and that were unique expressions of pupils’ emergent organisations of learning practices. 
These were the both ritualized and improvisational rhythms involved in enacting social-material bricolage as 
an aspect of learning. I shall give examples of pupils’ work below. 



Eigure 1 . Bricolaging in Geography 
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Figure 2. Translating from One Materiality Into Another 

This boy (Figure 1, on the right) is working on a poster about Iceland for geography. He is using Google 
to visualize the exact proportions and colours of the flag. He is engaged in a process of copying and 
translating the image of the flag onto a poster in order to present aspects of the subject Iceland to his 
classmates and teacher. For this pupil, the process of copying and translating is mediated through different 
materialities (cardboard, pen, ruler, iPad) in order to produce a third materiality, the poster (and flag). Part of 
this process is the enactment of a bricolage, where the different materialities are brought together in order to 
support the process of copying and translating from one materiality to the next. The role of the iPad in this 
bricolage is among other things to support and enhance the naturalistic visualization and reproduction of the 
flag. In this way the iPad contributes to stabilizing the pupil’s process of learning by copying and 
reproducing information through bricolage. 


Figure 3. Bricolaging in Maths 

This girl is doing a math assignment and is using the math book, her iPad, and a jotter to perform the task. 
She uses the book to check the tasks, the iPad as a calculator and her jotter to write down the results. Her 
work on the math task is therefore constituted through a chain of materialities which she combines in a 
bricolage in order to solve the tasks. 
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Learning processes involved in this bricolage is identifying the problem, calculating, and copying onto 
paper. In this enactment of bricolage the iPad enhances the process of calculating by being at hand and 
possibly providing a larger screen than the calculator usually used in math lessons. 

The two examples show how pupils use materialities to support processes of understanding, copying, 
enhancing, and visualizing through the enactment of bricolage. These processes serve to stabilize rhythms of 
learning that are well-known and ritualized in pupils’ daily work in the classroom. The enactments of 
bricolage are examples of students’ improvisations as they use what is at hand to support and enhance the 
learning process. These improvisations are continuous shifts in the ways in which pupils work, shifts that 
both stabilize and transform the ways in which they are learning. In this sense the enactments of bricolage 
become emergent transformative practices initiated by pupils themselves. 

5.2 Wider Circles: Extending Improvisations 

Though the enactment of bricolage and pupils’ improvisations are often used to stabilize rhythms of learning, 
pupils sometimes extend their improvisations to a wider array of materialities. These improvisations become 
more explorative and possibly transformative enactments of bricolage, where physical movement between 
materialities at hand becomes an added element, and where a wide range of materialities are incorporated. 

An example of a wider circle of improvisation is given below. 



Eigure 4. Working With the Solar System 


The girls shown in Figure 4 are working on a poster on the relationship between the planets in the solar 
system. This is an assignment given by the teacher, and essentially an assignment that engages the pupils in 
finding, copying, selecting and visualizing information - as in the examples given above. However, these 
girls choose to involve a wider array of materialities in their enactment of bricolage, than in the examples 
above. As their goal is to produce an accurate representation of the relationship between the planets, they 
start out by taking pictures with their iPads of a poster used by the teacher in the class. 
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Figure 5. Using Google and Pen and Paper 


In addition to photographing the poster the girls use Google to understand the constellations of the solar 
system through a different representation (a picture on Google), which they then sketch onto a piece of paper 
with a pencil (see figure 5). This corresponds to the enactment of literacies through circulations of materials 
described by Leander and Lovvorn (2006, see above), with the notable difference that the iPad provides a 
visual dimension to the processes of transferring and translating information from one material resource to 
the other. The visual dimension of knowledge acts as an added modality in the bricolage which helps the 
pupils to understand spatial organisations and dimensions of the planetary system. In addition to this, the 
iPad provides the pupils with knowledge that is not included in the authoritative material provided by the 
teacher (the book and poster). The representation of the planetary system found in Google for instance 
contains an extra planet, not shown on the teacher’s poster, which sets the pupils off on an added journey of 
discovery of information about the planets. 



Figure 6. Measuring for the Poster through the Picture of Another Poster 
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Following the processes in Figure 4 and 5 (googling, sketching and photographing) the girls expand their 
understanding of the solar system by using a compass to measure distances between and proportions of 
planets and to sketch them out onto their poster (see figure 5). After sketching they add colour to their poster 
and take a picture of it to show at home to their parents. Taking the photo with their iPad is part of the 
assignment, and another end project of the assignment is the poster which is supposed to be part of an 
exhibition at the school later that week. 

The process of producing the poster of the solar system is an example of an extended process of 
improvisation where a number of materialities are bricolaged through the pupils’ enactment of linkages 
between materials - for instance posters, paper, picture on Google, compass, pencils etc. The process of 
learning is therefore embedded in and enacted through materialities at hand, materialities that are combined 
to constitute the learning processes of these pupils. What is at hand in this example is defined both by the 
instructional trajectories of the teacher and the conceptual and physical combination of materials linked by 
these pupils, who use the iPad as a mediator for the use of different materials, modalities and activities in the 
learning process. Though measuring, copying and translating are still central processes in the learning of 
these girls, their improvisations are therefore more extended than in the examples described above, an 
extension that requires the physical movement of pupils between resources in the class, and their conceptual 
linking of an extended number of artifacts and modalities to constitute processes and end products. In these 
extended improvisations the iPad acts as a significant contribution to the emergent changes in practices - for 
instance by evoking flexibility, portability and multifunctionality. 


6. CONCLUSIONS 

In this paper I have argued that socio-material bricolage is a concept that can capture and encapsulate the 
emergent, practice-based rhythms of educational change that are associated with the presence of an always 
connected and accessible device like the iPad. Socio-material bricolage, I propose, enables us to understand 
how educational technologies are involved in and contribute to constituting practice over time as a response 
to teachers’ and learners’ shifting educational needs. The concept of socio-material bricolage is an approach 
to change that is in many ways at odds with a linear and evolutionary approach to educational transformation, 
such as that associated with the implementation of ICT in schools. I call the processes associated with socio- 
material bricolage improvisations in order to underline the shifting and unpredictable rhythms of change, and 
to stress the ways in which teachers and learners use what is available to them in specific places and times for 
their own purposes. IPads are central actors in these improvisations, I suggest, because iPads are at hand and 
can adapt to a diversity of learning needs, including different modalities of learning. Improvisational rhythms 
that include mobile devices such as the iPad may therefore support pupils and teachers in constituting 
learning processes and environments that are inclusive to their individual needs at specific times in their lives 
at school. 
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ABSTRACT 

With the increasing complexity of technology and large quantities of data in our digital age, learning and training has 
become a major cost of employers. Employee competence depends more and more on how quickly one can acquire new 
knowledge and solve problems to meet pressing deadlines. This paper presents a practical method to use REU (Research 
Experience for Undergraduates) projects and crowdsourcing to help students learn complex content as needed. The major 
question addressed in this work is to find ways to reduce the cost of learning for both the learners and their mentors. As a 
work-in-progress, we would like to share our preliminary design in using natural language processing (NLP) and data 
mining tools to retrieve and structure instructional materials semi-automatically. 


KEYWORDS 

Learn on demand, crowdsourcing, data mining, conceptual models, (Research Experience for Undergraduates) REU 


1. INTRODUCTION 

The digital age is characterized by big data, increasing complexity and rapid technology changes. Existing 
academic programs are inadequately structured to prepare students with the skills and knowledge needed for 
today’s complex fast moving work. Learning and training has become a major eost to employers and many 
small businesses are showing a tendency to hiring graduates with some work experiences but average 
academic records rather than hiring new graduates with above average academic with no experience. This 
brings great challenges for universities to place their new graduates. Research Experiences for 
Undergraduates (REU) has been advocated in recent years as co-curriculum activities to provide students 
hand-on experiences to complement the traditional undergraduate programs. Hands-on research experiences 
bridge the gap between problems in textbooks and real-world problems and therefore serve as a good 
stepping stone for our students in obtaining their first job. Student success upon graduation will greatly 
depend on how quickly one can learn new knowledge as needed and solve emerging problems. The purpose 
of REU programs is to (a) motivate students to learn new domain knowledge as they need it in order to solve 
problems and to (b) cultivate students’ problem solving ability. 

However, REU projects are time consuming for both student team members and faculty mentors. The cost 
for the learners can be measured by the total time consumed from the starting point of identifying missing 
knowledge to the end point of being able to solve the problem. REU is intimidating to most undergraduate 
students because it requires them to spend significant time acquiring knowledge beyond the standard 
curriculum for their major. The lecturing type of traditional curricula can be called as just -in-case education. 
It is designed to prepare students with well-refined knowledge and fundamental skills associated with a 
particular discipline. As co-curriculum activity, REU can be called learning-on-demand education. Its 
learning environment is more like that in industry rather than an academic setting. Its purpose is to teach 
students to acquire new knowledge and emergent technologies quickly in order to solve the problems at hand. 
Consequently, the students in REU projects are learning as needed and learning by doing so that they can 
significantly reduce the destructive effects of forgetting. This type of learning also enables the learners to 
take advantage of the ubiquitous internet memory to extend their biological memory (Starmer, 2012). 

The major cost for faculty mentors is the time involved in preparing and teaching the required domain 
knowledge needed to complete projects. 
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Because of the interdisciplinary nature of the REU projects, the mentor often does not have all the 
knowledge or the instructional materials on hand to solve the problems posed by the projects. In addition, 
mentoring REU projects is counted as services without teaching relief in most universities. Hence, REU is 
discouraging to faculty members. Our examples demonstrate a practical method to reduce the cost of learning 
for both the students and mentors. 

A primary role of a faculty mentor supporting a project team is identifying a student’s knowledge gap. 
The students should take the initiative to search and learn the knowledge from the internet, especially from 
the crowd sourced educational materials. While it is impossible or time consuming for the mentor to learn 
everything in advance and prepare the course material, the mentor may save time by focusing on 
understanding the relevant domain knowledge at the ontological level and building a conceptual model of the 
factors used to solve the assigned tasks in a logical and chronological order. 

The next section of the paper illustrates our system engineering approach and modeling tool Opcat to 
scaffold REU. The third section introduces the challenges and strategy for mentoring on REU projects. The 
fourth section presents work-in-progress for identifying and organizing educational courseware using 
educational crowdsourcing technology (Eytan, 2012, and Karger 2012). The fifth section describes future 
work using GATE, an NLP tool and Weka, a data mining tool, to retrieve and extract instructional materials 
for mentoring students engaged in REU projects. 


2. USE SYSTEM ENGINEERING APPROACH TO SCAFEOLD REU 

Liu and Ludu (Liu 2012) identified three major challenges that hinder RE success: 

1) Critical thinking: Many undergraduates just plug data into formulas to solve problems and show 
little ability to extend mathematics concepts beyond an algorithmic level. 

2) Complexity: Undergraduates lack the experience and knowledge to divide complex problems into 
multiple small problems and the complexity of the research is beyond their current knowledge. 

3) Applicability: Undergraduates have limited domain knowledge. Therefore, they cannot apply the 
knowledge to validate their mathematical models. 

To address these shortcomings, Liu and Ludu established a system engineering-based REU program 
called ACE. Shown in Eigure 1, ACE stands for Analysis, Computation, and Experimentation. Liu and Ludu 
have also established a nonlinear wave lab facility and Eco-Dolphin project to provide students a hands-on 
development environment for the validation and verification of objects they have designed and developed in 
their projects. 



Figure 1 . Opcat Diagram of the ACE Paradigm 

We have selected Object-Processing methodology and its support tool Opcat for building conceptual 
models. Opcat is a user friendly tool with a gentle learning curve (Dov Dori, 2002). 
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Opcat offers dual representations; Object-Process Diagrams (OPD) and language (OPL), which enables 
an Opcat model to automatically include the graph legend. Opcat uses three entities as building blocks: 
objects, processes, and states. Objects are things that exist. Processes are things that transform objects by 
changing their states and creating or consuming objects. In OPD, objects, processes, and states are 
symbolized by rectangles, ellipses and rounded rectangles, respectively. The single integrated view of the 
Opcat models and the built-in syntax of Opcat based on OPM methodology help modelers identify missing 
components and breaks in logic links. The zooming in/out strategy for complexity management makes it easy 
to build and understand the conceptual models at almost any level in a hierarchy of a system and at different 
levels of granularity. These features make Opcat an ideal tool for quick prototyping and requirement 
gathering. The conceptual models help users to discover the critical paths at a project management level and 
identify the concepts, knowledge, and skills that a team needs in order to complete the project (Liu 2004). 

Our system engineering approach shown in Figure 2 and 3 guides students to take small steps, analyze the 
problems incrementally and refine the results iteratively. The two Object-Process Diagrams have the same 
iterative elaborating loops for sense making (conceptual model) at the qualitative level and validating the 
mathematical model at the quantitative level. The data driven model validation is based on sound scientific 
principles. This approach can provide evidence based assessment for student learning. Students can evaluate 
their own solutions to problems without the teachers grading their work. Hence, our system engineering 
approach saves mentor’s time by reducing the time needed to evaluate student work at the detailed level. 




Eigure 2. Sense Making for Conceptual Model 



Eigure 3. Validating for Mathematical Models 
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Table 1. Computer Generated Sentences for OPL for OPDs in Figure 2 and Figure 3 


Sense Making for Conceptual Models 

Conceptual Model can be Confirmed, Invidentififed, or 

VRecognized. 

Confirmed is final. 

VRecognized is initial. 

Factors Principles can be FactsCollected, AssumptChecked, 
or FactsConfirmed. 

FactsCollected is initial. 

FactsConfirmed is final 


Validating Mathematical Models 

MathModel can be Validated, Elaborated, or Recognized. 
Validated is final. 

Recognized is initial. 

Data can be Collected, Filtered, or Processed. 

Collected is initial. 

Processed is final. 

DataProcessing requires Elaborated MathModel. 
DataProcessing yields Processed Data. 


3. LEARNING-ON-DEMAND STRATEGY FOR REU PROJECTS 

The strategy to help undergraduate students employ learning-on-demand methods is to divide the REU 
projects into components (subsystems and tasks) at the proper level of granularity. The division of 
components not only needs to fit the nature of the problem, but also the academic requirements of team 
members. Figure 4 depicts the conceptual design of Eco-Dolphin project under the mentorship of Liu. Eco- 
Dolphin is the name of a fleet of adaptive and cooperative Automated Underwater Vehicles (AUVs) that a 
team of 6 SIAM Chapter students at ERAU have been working on since spring semester 2012. It will be 
designed to support future environmental science research and surveillance services in littoral water. Eco- 
Dolphin serves as a component of the ACE program (Liu 2012) that is sponsored by the department of 
mathematics and Honors Program at ERAU. The autonomous nature of robotics spans multiple disciplinary 
fields including mathematics, computer sciences, physics, mechanical engineering, electronic engineering, 
computer and software engineering. Consequently, the Eco-Dolphin project is a truly “trans-disciplinary” 
REU program. The 6 team members come from 6 different degree programs and are under the mentorship of 
three professors, whose expertise covers mathematics, physical sciences, computer sciences, software 
engineering, and Electronic Engineering. The team also includes a graduate student mentor in Mechanical 
Engineering. 



Figure 4. Top Level of System Conceptual Models of Eco-Dolphin Project 

The interdisciplinary nature and hands-on REU project requires each member of the team to learn 
knowledge relevant to a wide range of assigned tasks. Since it is not feasible for a faculty mentor to teach 
each team member on a one-to-one base, the mentors and students have collected hundreds of relevant 
technical articles and organized them according to the subsystems named above. The files are shared by team 
members through Dropbox. 
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Our projects require mentors to learn more new knowledge than any student. For example, a robotics 
project typically starts as a mechatronics project and ends as a software engineering project. However, a 
faculty mentor may not have experience in either mechanical engineering or electronic engineering. To 
address this problem, we implemented a learning as-needed strategy by starting a literature review and 
identifying a few AUV projects as peer projects that the Eco-Dolphin team would emulate. The mentor 
tracked the references from the student reports and then traced the references on every subsystem. The other 
two faculty members served as advisors to the lead mentor of the project. 

The OPL sentences in Table 2 help the modeler determine if the diagram in the OPD in Figure 4 confirms 
one’s intention. The dual representation of the Opeat model faeilitates communieation among teammates and 
serves an idea tool for brain storming. 

Table 2. Computer Generated Sentences Are a Sample OPL Associated with the OPD in Eigure 4 


System decomposition 

Eco-Dolphin System consists of Ground Station and 
Eco-Dolphin Eleet. 

Ground Station is physical. 

Ground Station consists of DB-Pilot Position 
System4Ground, AcouComPort, and WIFIComPort. 

DB-Pilot Position System4Ground is physical. 


AUV consists of Ballaster, 3 to 6 Propollerses, 
many SafetySensorses, and many Navigation Deviceses. 


Component dependent relationship 
Actuating requires AutoPilot System. 
Actuating yields either Propollers or Ballaster. 
Sensing requires SafetySensors. 

Sensing yields Safetyinfo. 


WayPoint Setting requires Superving Symstem. 
WayPoint Setting yields AutoPilot System. 
Positioning requires Navigation Devices. 
Positioning yields Superving Symstem. 


After the mentors collected references, the student team leaders took the initiative to study the required 
knowledge associated with their assignments. The faculty mentor then prepared video lectures and online 
instructional materials for difficult components such as PID (process, integral and derivative) control and the 
Kalman filter needed for the navigation program and architecture design of software systems. In the first 
phase, the team successfully designed and built the hull. It includes the ballast, hydro pressure sensors and 
propeller, etc. But, the software component has not met the expected goal because of a lack of programmers. 
As the project progresses, the students and faculty mentor have to constantly learn new knowledge as needed. 
We have found that faculty learning time can be reduced by letting students lead the way. For example, 
students stay ahead of the faculty mentor in a new technology such as the Ardruino microcontroller 
programming and tutor each other. 

The success of mentoring REU depends on assigning small tasks to students and frequently meeting with 
them to discuss issues in completing their tasks. While graduate student research advisors can assign tasks 
and meet them monthly, the mentors of REU assign tasks and meet students every week or every other week. 
However, the mentors of REU can save time by finding crowd-sourced materials and enabling students to 
learn on their own or from each other, instead of teaching traditional classes. 


4. EXTRACTING MATERIAL EROM CROWDSOURCING EOR 
LEARNING ON DEMAND 

Crowdsourcing is the process of assimilating many small contributions into resources of high-quality. Khan 
Academy, MIT Open Courseware, Stanford online Courses, Udacity, and Coursera have made a 
transformative impact on the “digital generation’s” ehoice of college education and methods to gain 
knowledge. The scope of their collective impact and the influence on the infrastructure and subsequent 
impact on college education (e.g. virtual open universities) are too significant for any university to overlook. 

Free courseware can also be exploited as supplementary education resources to provide students 
personalized education and support learning on demand. In addition, it can save a significant amount of time 
for mentors of REU projects to prepare instructional materials. In most subject fields, there are adequate 
learning materials from crowdsourcing. In order to leverage this free courseware, three major issues need to 
be addressed: 
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1. How to automate the process of identifying and extracting the online materials. 

2. How to track the dynamics of web updating and trace breaking links. 

3. How to evaluate the quality and the appropriateness of materials based on the needs of learners. 

The authors are currently working with student research assistants on a project to build an Open 

Courseware Identification Subsystem (OCIS), which is expected to be completed in the summer 2014. The 
OCIS project has set two goals: the first goal is to collect calculus course materials that are compatible to our 
syllabi as complementary tutorial material; the second goal is to explore the information technology to 
automate the data retrieval and information extracting process. Collecting open courseware for a few courses 
manually is more efficient than developing a web application to automate the process. Consequently, the 
authors have manually collected computational science education materials online for a course in 
Mathematical Modeling and Simulation. However, the manually implemented process neither scales to large 
set of courses due to the prohibitive labor cost nor supports the dynamic content change of the online 
courseware. 

Instead of writing our own natural language processing program, we chose to use a NLP tool called 
GATE (General Architecture for Textual Engineering) to fulfill the second goal mentioned above. GATE is 
to NLP as MATLAB is to numerical computation or Mathematica is to symbolic computation. Novice users 
can apply GATE’S built-in resources to process common tasks such as name recognitions in the same way as 
novice MATLAB users call several MATLAB commands to perform matrix operations. Advanced GATE 
application developers may need to write GATE programs to process complicated tasks such as identifying 
and summarizing the particular newsletters of interest to their sponsors. A GATE application includes three 
components: language resources, process resources and an application pipeline made of built-in processing 
resources, third part plug-ins or user made plug-ins. Besides the corpus in data store, the GATE users 
typically need to create another language resource called an annotation schema. There are many options in 
GATE. It can be either a simple gazetteer with the keywords of a domain or an ontology driven gazetteer 
with logical relationships of the keywords. The user inputs one’s domain knowledge into the system from the 
gazetteers. Eigure 5 shows the flowchart of the processes for a web mining and KDD system (Wang 2012). 
Eigure 6 gives a different view of such a flowchart for a particular project (Wang 2013). Eigure 7 gives the 
detailed flowchart of the IE subsystem on the left side of Eigure 6. 



Figure 5. Flowchart for Web Mining and KDD, Figure 6. Web Application to Graduate School Matching 



Figure 7. Information Extraction (IE) Framework for Education Web Mining. 
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We will not completely automate the Information Extraction (IE) processes for OCIS because it is 
difficult to connect several tools from different vendors in a pipeline. Instead, we will manually launch each 
tool in the pipeline and exploit the strength of the tools one at a time. The IE in Eigure 9 has three steps: 

Step 1, we use xml-sitemaps - to download the website maps of courseware providers such as Khan 
Academy or the MIT Open Courseware. The free version allows us to crawl and download up to 500 pages 
each time. To download the sitemap of MIT open courseware, we input it intoxml-sitemaps. 

Step 2, we use GATE to process the sitemap document and extract the links of interest. We first save the 
loaded files to the data store of GATE and then create a gazetteer with the keywords of the calculus courses. 
Next, we create a GATE application by pipelining the ANNIE processes. The output of this GATE 
application is the links of the courseware that we provide to our students. We can run the GATE application 
once a semester to check for new updates and break links autonomously. 

Step 3, we will check the quality and appropriateness of the website. Google webmaster can be used to 
find the download rate and other properties of the websites. We will read the reviews for the small number of 
websites in the OCIS project. Einally, we randomly select sample content of the selected courseware and 
send it to colleagues and student assistants to review for quality and appropriateness. The links and property 
summary of websites as shown below will be saved to our database. 

Address = http://www. youtube. com/watch?v=AUqeb9Z3y3k&feature=relmfu 
Keywords_ in_link = {Mathematical, Modeling, Simulation} 

ErequentlyUsedWords = (Matrix, Transformation, Scarce, toeplitzj 
Download rate = 98234, number of reviewers = 57 
Document_properties = (6368986 Bytes, Joe John, 2008/12/01, 2010/4/05} 

As a relevant previous project, Eigure 6 shows the flowchart for a web application for matching graduate 
schools for graduate seniors in colleges. The shaded half on the right hand is the Graduate Research Project 
(GRP) completed by Wang under the mentorship of the first author in 2012 (Wang 2013). 

In addition to developing the web application, the major theoretical component of the GRP project is to 
use the Weka data mining platform a to analyze several data mining algorithms based on the performances of 
the graduate school matching problem. The GRP covers the data mining component and OCSI covers the IE 
component of figure 6. Combining the two projects, allows us to cover all the technology described in Eigure 
6. Ideally, we should have used the same application to cover the entire system in Eigure 6. However, we 
made the choice for the projects based on the interests of the graduate student and the sponsor of OCIS. 
Many components are reusable for future work because the GRP project, OCIS, and the information 
extraction for REU projects are similar educational web mining applications (C. Romero, 2007). 


5. FUTURE WORK 

As the OCIS project continues, we would like to use data mining to match the open courseware stored in 
database and the syllabi of our corresponding courses. The task is to complete the shaded steps 4 and 5 on the 
right side of the Eigure 6. If we replace the profiles of graduate schools by the content tables of the open 
courseware and the graduate applicant forms by the syllabus (including the topics for each lesson) of our 
corresponding course, the data mining application is almost identical to the GPR project mentioned above. 
The output will be the matching of content between the open courseware and the lessons of our 
corresponding courses. We are doing this matching manually for the OCIS project because the scope exceeds 
our one year time frame. Our next application is to semi-automate the process for finding open education 
materials associated with our particular REU projects. The major difference between finding courseware for 
the known courses and materials for REU is that the logic order of the former is well known in standard 
curricula and that of the latter is most likely unknown. 

Another goal is to automate the process to obtain structured user review summaries from unstructured 
online user reviews. GATE can be used to obtain semantics level recognition (e.g. name recognition) easily. 
GATE developers usually need to write Java script to recognize hidden relationship of words based on 
contexts for summarizing documents. These patterns are unlikely to be recognizable at the semantic level. 
Eor example, the two OPLs shown in Table 1 for mathematical model validation and sense-making 
conceptual models have the exact same structure and logic relationship. However, shallow semantic level 
comparison cannot identify the two diagrams because there are no identical words used in the two diagrams. 
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Nevertheless, the OPL in regular language contains a clear pattern based on the few syntactical relationships 
of three entities (object, process and states). Hence, the ontological model in Opcat can be used by the GATE 
program to identify hidden relationships based on structure similarity between the training corpus and 
targeted corpus. GATE actually supports ontology driven unsupervised annotation. 

This paper addressed the problem of how to motivate students to learn as needed by providing them 
hands-on research projects and giving them the opportunity to learn by doing. A system engineering 
approach called ACE is introduced to help students evaluate their own research solutions. Conceptual 
modeling and its support tool, Opcat, is advocated to scaffold learning of complex systems. We offered the 
first hand experiences about how to take advantages of crowdsourcing education materials to save time for 
preparing learning materials. While the traditional just-in-case education requires students to memorize 
formulas, definitions and methods to pass tests, just-in-time learning and learning by doing can significantly 
reduce the destructive effects of forgetting and let the learners take advantage of the ubiquitous internet 
memory to extend their biological memory. A major challenge for faculty mentors of REU projects is the 
time consuming tasks to prepare learning materials out of the main stream curricula. We shared the designs 
of our previous project, an ongoing project, and future work about how to use natural language processing 
tools and data mining tools to find quality and relevant open educational materials online efficiently. 
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ABSTRACT 

This paper builds on data from a project where Ipads were used in a lower secondary school in Denmark to support 
school development and inclusive learning environments. The paper explores how iPads enter into and work as part of an 
ecology of learning in five classes in lower secondary school. The paper argues that we should disengage approaches to 
the iPad in education from ideas of what the properties of these technologies are, and see the device as a more relational 
and situated actor, avoiding the definition of properties of technologies outside the contexts specific to their use. 
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1. INTRODUCTION 

Though iPads were originally intended for consumption and entertainment, they are increasingly used for 
learning in formal education. This brings new challenges and potentials to classrooms, where the use of 
technology has often been associated with computer labs and with Teaming in a bubble’ (Traxler 2010, 5). 
With the advent of the iPad technology is increasingly taking center stage in the daily life of pupils, including 
how they can learn, interact and create content in formal educational settings. The promise of the iPad 
therefore seems to be the liberation and transformation of education at a time where mobile technology is 
defining most out of school activities of young learners. Consequently, the so-called affordances of the iPad, 
i.e. mobility, intuitive navigation and personalized content creation, has been the focus of many accounts of 
how the iPad can contribute to education and learning (e.g. Burden et al 2012). 

However, though the tablet seems to be an innovative and promising platform for 2C‘ century learning 
environments, discussions about its affordances should focus on educational practice and not its ‘inherent’ 
qualities. This paper will argue that tablets, as emergent mobile devices, may contribute to enhancing 
inclusive educational settings by creating new relationships between existing technologies and learning 
resources for mobile learning. A general ‘affordance’ of the iPad therefore seems to be its flexibility and 
ability to enter into relationships with a variety of resources and learning contexts that make up a learning 
ecology for schools. 

The paper builds on data from a project where iPads were used in a lower secondary school in Denmark 
for school development. The research discussed in the paper focuses on how the integration of iPads in 
teaching and learning can support inclusive educational settings in lower secondary schools. The paper draws 
on data from fieldwork in five classes of seven graders (age 13-14), who were given iPads in the school year 
2012-13. Two of these classes were special needs classes. 


2. IPADS IN EDUCATION 

Since the iPad was introduced on the market in January 2010 it has received extensive attention for its role as 
a “game changer” (Furfie 2010) in the media field as well as in education. iPads can in this regard be 
compared to other mobile devices that are increasingly transforming the ways in which we access and create 
knowledge, communicate and collaborate and learn (Pachler 2007). 
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However, contrary to other mobile technologies, iPads have generally been admitted into educational 
settings, where they have become central to learning and to the development and transformation of schools. 
In this sense iPads promise to ‘revolutionise’ education. 

Though extensive empirical knowledge about the value of tablets is still lacking, several recent studies 
have confirmed the educational value of the iPad at different levels of education (Burden et al. 2012, 
Melhuish & Faloon 2010, Heinrich 2012, Kinash, Brand, Mathew 2012). Generally, these studies argue that 
the significance of the iPad for teaching and learning resides in two significant affordances of the device, i.e 
1) providing new forms of personal ownership and 2) ubiquitous and easy access to technology. 

Ownership is central to learning in that it allows the user to personalize devices and to support learners’ 
own knowledge and conceptual frameworks (Melhuish & Falloon 2010). Mobile devices may in this sense 
support ‘constructivist’ learner-centred approaches to learning and be useful to young learners who are 
already immersed in technology through every day uses of for instance smartphone devices. In addition to 
this the use of app-based teaching promises a shift from content based, skill and drill learning to web 2.0 
approaches to learning where the learner is more creative and independent. Finally, the low tech, intuitive 
and multimodal feel of the iPad targets a variety of learners and learning styles, which may support more 
inclusive classrooms and learning environments. 

Ubiquitous access to technology may likewise revolutionise education as it allows teachers and learners 
to redefine learning spaces in moving technology use from confined, fixed places and times to situated, just 
in time usage, where technology is “Woven into all the times and plaees of students’ lives” (Traxler 2010, 5). 
Thus, Burden et al (2012) argue that the shift in technology use in schools linked directly to the allocation of 
the iPad ean be eharaeterized as a shift from a ‘just in ease’ model where teehnology is made available from a 
remote location from the loeation itself to a ‘just in time’ model where teehnology is on hand, immediately 
accessible to the pupil. 

Though the affordances of the iPad accounted for and explored by the studies mentioned above are 
highly relevant in understanding how iPads and other emergent mobile technologies can support education, 
the idea that iPads are isolated and unique actors in school development must and can be challenged. A 
recent study of the use of iPod touch devices in primary education in Australia thus underlines that in many 
cases teachers integrate mobile devices with other ICT technologies such as desktop computers and laptops, 
Nintendo Wiis, digital cameras, podcasting software, video editing suites, etc. (Murray & Sloan 2008). The 
mobile device in this way emerges as one of a range of tools that the teachers employ to motivate and 
stimulate student learning. Similarly, Burden et al. argue from their experience with the use of iPads in 
primary and secondary education in Scotland that “results suggest students use the device as part of a wider 
ecology of learning resources, integrating the iPad with existing tools such as the jotter” (2012, 51). If we are 
to understand the full potential of the iPad for teaching and learning, I therefore propose that we see the 
device not as an isolated actor in school development but as a more relational and situated actor, i.e. one that 
is “bound up with the specific material-discursive practices that constitute certain phenomena” (Orlikowski 
2010). A relational and situated understanding of the iPad will attempt to avoid defining the properties of 
technologies outside the contexts specific to their use, contexts in which they contribute to constituting how 
learning can be enacted. 


3. IPADS AND SCHOOL DEVELOPMENT 

In Denmark, so-called ‘iPad-schools’ have become a growing phenomenon in the field of school 
development across the country. A number of schools and municipalities are thus investing in iPads on a one 
pupil one device basis. Earlier this year a municipality in the west of Denmark invested in iPads for all pupils 
and teachers in the municipality. Other municipalities and schools have followed, however, most schools 
have opted for a less costly investment, by for instance focusing on buying tablets for specific groups of 
learners or teachers. Therefore, ownership models may vary in different schools, and even within schools. 

What seems to be the argument for investing in tablets for school development is complex, in that schools 
are seeing technology both as a way to improve their economy in a time of recession, a way to enhance the 
profiles and reputation of especially state financed schools and an approach to transforming teaching and 
learning in classrooms and beyond. 
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Economic considerations usually focus on the fact that ubiquitous technologies can help schools save 
money on resources such as paper copies and books, and that iPads require less maintenance than other 
technologies. In terms of school profiles and reputation the move in Denmark (as well as in other parts of the 
world, see for instance Anderson-Levitt 2003) towards decentralization has made it more urgent for schools 
to attract sufficient numbers of pupils by marketizing their pedagogical visions and principles. In this context, 
the use of iPads in education may contribute to connecting schools with 21“" century approaches to learning. 

In relation to education, a general political focus on basic education has underlined the need for school 
reform and for increased access to learning resources that can support the learning needs of different learners. 
A lot of political interest in Denmark has thus lately been focused on primary school and pre-school levels, in 
order to create more continuation between these levels of education, and in order to bring more learning into 
preschool levels, to prepare students for formal learning (Jensen, Brostrom & Hansen 2010). In formal 
learning, there has been an increased pressure on pupils’ literacies in for instance Danish and Maths and in 
assessing pupils’ competences at different levels of schooling. The role of iPads in this educational 
environment is, it can be argued, to support the general reform of schooling in making technology more 
accessible to students and more integrated into the everyday life of schooling. Giving tablets to children 
according to a one child one tablet principle is for instance thought to increase the engagement and 
participation of children in learning. In addition to this, mobile and personalized tablets support, it is argued, 
the transformation of learning spaces that will allow schools to be more inclusive of different learners and 
learners’ needs, including children with cognitive challenges. These are some of the contextual realities for 
the research described below. 


4. MOBILE LEARNING IN AND OUTSIDE THE CLASSROOM 

Middletown is a lower secondary school in the west of Denmark in a municipality that has a high profile in 
school development and integration of ICT into education. The school has recently been through a process of 
merge where pupils from an associated school for children with special needs were integrated into the school. 
The school has not had a prominent ICT profile before the project, mostly due to budget restrictions. 

The school teaches pupils at three levels, i.e. 7*, 8* and 9* year of schooling. Pupils come to the school 
from other schools in the area, and it is therefore important for the school to accomodate pupils from 
different neighborhoods and backgrounds. 

At the beginning of the school year (2012) all pupils in the 7* form (3 classes) as well as two special 
needs classes were given iPads to keep for the entire school year. Teachers in the seventh form were given 
iPads before the summer holiday, so that they would have time to explore the tablet before using it in classes 
with pupils. The municipality had decided that this initiative should be followed by research, in order to 
investigate the role and learning potential of iPads at this level of schooling. The research was aimed 
specifically at understanding how tablets can support the inclusion of pupils within a variety of learning 
environments and subjects, as inclusion is a challenge that is currently at the center of policy at both 
municipal and state levels. In this project inclusion is understood as a broad concept, i.e. with a focus on 
inclusive educational settings where all pupils are valuable and active participants in the learning community 
(Tetler&Baltzer2011). 

I followed pupils in all five classes for 3 months, observing them in their daily life in school and 
interviewing groups of pupils and teachers as well as the school leader in the process. 

At the time when tablets were distributed to teachers, technology was, as mentioned above, not a widely 
used tool in the daily life of the school. What was available to pupils and teachers at this school was 
primarily two computer labs in the basement of the school as well as whiteboards in all classes. When the 
school decided to invest in iPads for the seventh grade pupils and teachers, it was however necessary to 
install wi-fi in major parts of the school, which immediately enhanced teachers’ and pupils’ access to the 
internet. The investment in iPads therefore initiated something the school had wanted for years, i.e. the 
opportunity to integrate technology on a more general basis in teaching and learning. The iPads therefore 
became significant actors in moving school development in the direction of a more innovative and ubiquitous 
use of technology. 

In Middletown school both teachers and pupils were excited about the new technology and were open to 
the many ways in which it could be used in different subjects and different learning contexts. 
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However, knowledge of how the iPad can be used for education takes time, and has been mostly 
experiential for both teachers and pupils in this school, though teachers did have two courses in using the 
technology and using relevant apps before starting the school year with the pupils. This means that use of the 
iPad has to some extent been adapted to existing ways of organizing learning and that transformation of 
teaching and learning has been strongly linked with having the technology available in classrooms and at 
home. I shall proceed to describe how personal ownership and ubiquitous access influenced the ways in 
which teaching and learning were done and to some extent transformed during the three months that I was 
doing fieldwork at the school. 


5. AFFORDANCES AND TABLETS IN THE ECOLOGY OE LEARNING 

IPads can be conceptualised as technologies with specific affordances, i.e. as proposed by Melhoon and 
Fallon (2010), portability, affordable and ubiquitous access, situated, ‘just-in-time’ learning opportunities, 
connection and convergence and individualized and personalized experiences. However, as mentioned above, 
these affordances should be understood in the context of how technologies are embedded into specific 
practices of teaching and learning and how these contexts integrate different kinds of learning ressources.One 
of the things that characterizes teaching and learning in Middletown School is for instance teachers’ use of a 
variety of resources for supporting learning, e.g blackboards and whiteboards, books, paper, cardboard and 
colouring pencils, rulers, calculators etc. In this sense IPads are rarely singular and isolated learning tools 
used in learning, but take on meaning from - and give meaning to - a variety of practices in which it is 
entangled. A general ‘affordance’ of the iPad is therefore, I shall argue, its flexibility and ability to enter into 
relationships with a variety of resources and learning contexts that make up a learning ecology for pupils’ 
learning. Below, I shall give examples of how the iPad is entangled with practices that constitute the learning 
potentials of this technology. 

5.1 Classroom Resources and IPad Usage 

IPads are, as argued above, often seen as transformative technologies that replace or marginalize other 
technologies in order to redefine learning spaces and reform teaching and learning. However, as I followed 
teachers, pupils and iPads into the classrooms of Middletown school, it became immediately obvious that the 
iPad would have to make a place for itself in a space where many different learning materials and media had 
historically been significant for practice. In this sense the iPad was neither entering an empty space, nor 
entirely replacing tools that had been used for decades for different kinds of subjects. 

Apart from the above mentioned whiteboards, what was significant for teachers was for instance to use 
books and paper (for teaching Danish and literacy), jotters, rulers and calculators (for maths), flasks and 
burners (for chemistry) and maps (for geography). These learning materials were not easily replaced by the 
iPad, though some of them might change their function over time, as pupils and teachers became more 
familiar with the technology. For instance, many pupils quickly learned to use their iPads to take notes, and 
therefore made some uses of jotters superfluous. Similarly, some teachers insisted that pupils should pick up 
and hand in their assignments through for instance Dropbox, activities that might change the ways in which 
paper and paper copies were used. 

Though the presence of the iPad in the classroom did therefore generally change and redefine uses of and 
relationships between used learning tools and resources, my observations also showed that teachers and 
pupils persisted in using a number of different learning materials with their iPads, implying that the tablet 
technology had not replaced but rather interacted with other learning tools and resources. Pupils for instance 
often connected the use of their iPads for various kinds of learning with checking their books, copies that the 
teacher had given them or even using calculators or pencils. In fact, pupils were in a number of cases 
assembling their personal combination of learning materials, when they were working on assignments. In 
these combinations of learning resources the iPad often had a central position as a tool that would allow them 
to for instance read tasks that the teacher had posted in Dropbox, check Google or record German 
vocabulary. I call these personal assemblages and combinations of learning materials installations, and will 
provide some illustrations of how pupils constructed these below. 
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I am arguing that these installations, in which pupils explore the relational affordances of the iPad and its 
multifunctionality, are significant for the ways in which pupils and teachers can use iPads for teaching and 
learning. 



Installation 1. Using the iPad with paper and colouring pens for geography 


This boy was very particular in getting the facts right when doing a collage for geography about Ecuador. 
He used Google for checking the colours and patterns of the flag, which he then translated into his own, 
naturalistic expression on cardboard, using colouring pencils and supporting images with pencil written 
textual explanations. The interaction between iPad usage and cardboard usage in this way allowed him to 
both find and understand information about Ecuador, and translate and organize his knowledge onto a 
different material, the cardboard. 



Installation 2. Using the iPad for Project Work 
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For this girl, learning about geography became a process of finding and selecting relevant facts about 
Iceland to present on her poster. She used her iPad to access Google and Wikipedia and focused on reading 
through information which she then copied onto her poster. Like the boy above, she was essentially engaged 
in processes of understanding information and then translating and copying it onto a paper display, her own 
expression of knowledge and content. Her use of the iPad together with paper and pen allowed her to access 
different kinds of information about her chosen theme as well as to translate it into something that could be 
presented to the class. 



Installation 3. Using the iPad for Learning German 

These two girls were collaborating on practicing German sentences by asking each other questions in 
German. They used their iPads for writing down vocabulary and for recording their pronunciation of the 
sentences in the PuppetPals app. They used their books to check spelling and vocabulary, as the teacher had 
asked them to practice specific areas of vocabulary in the book. They constructed their unique combination 
of books and iPads to be able to check information, record, listen and write down while they were working 
on their task. 

It can be concluded from the examples above that in a number of cases pupils built their unique 
installations, i.e. relationships between iPads and other resources such as books, pens and paper, to find and 
understand information, copy and translate it into their own context and in turn produce their own 
presentations of the knowledge, for instance as German sentences or geographical area knowledge. The 
processes involved in constructing these unique installations of resources for learning were therefore about 
translating, processing and disseminating knowledge. In building these installations the iPad acted as a 
flexible technology in terms of both size, form and functionality, allowing pupils to for instance use it as a 
tape recorder, a jotter, a dictionary, a display etc. In this way the iPad became part of pupils’ emergent and 
relational uses of different kinds of resources that were relevant for their specific learning needs. 

5.2 Whiteboard to IPad: Small Screen To Big Screen Relationships 

As mentioned above Middletown school is a school that for a long time has relied on whiteboards, computer 
labs and occasionally pupils’ personal laptops to support teaching and learning. These technologies have to 
some extent been connected with a more fragmented, occasional, use of the technology. 
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The iPad project promised an immediate change in the occasional use of technology in the school 
transforming technology use from a ‘just in case’ approach to a ubiquitous activity by supplying teachers and 
pupils with respectively a stable wireless connection and the portable, personal technology of the iPad. 

For teachers and pupils the iPad project was generally an opportunity to integrate technology on a more 
daily basis into teaching and learning as well as making connections between school learning and out of 
school learning and entertainment. Pupils’ choice of apps and other personal resources such as photos and 
desktop images would for instance illustrate their entertainment and leisure time preferences, family relations 
etc. - in the same way that they would generally use their smartphones for easy access to social media, games 
etc. out of school. 

However, in the case of Middletown School the move from whiteboards, notebooks and computer labs 
was a more complex situation than could initially be anticipated by the transition to ‘just in time’ approaches. 
First of all, whiteboards and other kinds of resources remained in the space where pupils were learning, i.e. in 
classrooms, and some pupils would still prefer to use their laptops or other resources for reasons explained 
below. Also, on occasion the computer lab would have to be used for printing out material that could not be 
printed from the iPad itself. In effect, what had appeared was not a new situation where iPads and mobile 
technology had entirely replaced prior technologies, but a situation where the availability of technologies had 
multiplied and new relationships had been established between ‘old’ and ‘new’ uses of technology. These 
relationships created new opportunities for teaching and for tailoring learning processes to individual pupils, 
i.e. for inclusive educational settings. 

It can be argued that the presence of iPads in the classrooms of Middletown School to a great extent 
moved the use of technology from the bounded space of the computer lab into classrooms, but that the 
presence of iPads in the classroom did not necessarily make the use of other resources in the classroom, such 
as e.g. the whiteboard superfluous. What emerged from this situation where a new technology had found a 
place in the classroom was therefore not a replacement of existing resources by a new technology, but a 
novel and possibly innovative relationship between resources such as for instance the whiteboard and the 
mobile technology. The whiteboard was one of the significant technologies involved in this new relationship, 
because the whiteboard had been available to teachers for some time, and because it was placed in a 
dominant position in the classroom and generally acted as an integrated tool in many teachers’ planning and 
classroom performance. In effect, the role of the whiteboard was largely maintained in the classroom when 
iPads entered the learning space. 

Whiteboards are to some extent tools that support the role of the teacher in the classroom, and situate the 
teacher as the authority of the learning space and of disseminating knowledge (Jensen 2010). In the special 
education classes in particular, teachers had been accustomed to using the whiteboard as a point of reference 
and connection in the classroom, where different kinds of relevant information, presentations, multimedia 
etc. could be displayed. The teachers would for instance use the whiteboard to display tasks that all pupils 
had to solve, show films and websites and summarize discussions. 

According to a teacher in one of the special education classes, the whiteboard was a good tool for 
focusing students’ attention on tasks, help them memorize and give instructions for assignments. In this way 
the big screen could support teachers in managing curriculum activities and assessment. However, this 
teacher also told me that for some pupils it might be difficult due to cognitive challenges to keep track of and 
focus attention on what was going on on the big screen. These pupils had, prior to the introduction of the 
iPad in the school, often worked on their own or with the teacher on their laptops where they could work on 
assignments in their own pace and for instance have text read aloud to them by software on their device. 

When the iPad entered this classroom it however became evident that the iPad could contribute to making 
access to the internet and the relevant software much more easy for the pupils, and on top of this that the 
tablet could act as a personalized small screen for pupils who had cognitive and other kinds of challenges. In 
the class I observed how for some of the pupils it would be useful to sit with the teacher or on their own and 
use the iPad as a smaller screen that could help them learn in a more self-directed way. The tablet could be 
used as a personal screen, the pupil’s own screen or a screen that could be shared between the teacher and the 
pupil. In this way the smaller screen helped pupils and teachers to display, interact with or produce relevant 
knowledge. 

This is not to say that teaching in this class took place primarily as an activity where the whiteboard 
would dominate and the iPad would act as a supplementary tool to the whiteboard screen - very often it 
would be the other way around, or the big screen might not be used at all. 
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Sometimes pupils would for instance produce presentations on their tablets that would then be displayed 
from the iPad onto the whiteboard, in order to share with the class. At other times, pupils would get their 
instructions for assignments from Dropbox or from the internet rather than from the whiteboard screen. What 
emerges from this analysis of how iPads were integrated into the classroom is, that teachers and pupils were 
able to use the technologies available to them in ways that made sense to them for specific learning purposes 
and contexts at specific times. There is no doubt that the advent of the tablet enhanced the learning processes 
in this classroom considerably, for instance by providing pupils with a personal device that could support 
them in producing context, accessing information and managing tasks. However, the experience that emerges 
from these activities is that the presence of the iPad generally enhanced relationships between teachers and 
learners, between learners as well as between learning resources available to teachers and learners rather than 
acting ‘on its own’ as a separate device in teaching and learning. 


6. CONCLUSIONS: EMERGENT TRANSEORMATIONS AND MOBILE 
TECHNOLOGIES IN THE CLASSROOM 

In this paper I have argued that research in the educational value of iPads can be qualified by understanding 
their situated contribution to learning, i.e. the complexities of how the technology is embedded into the 
contexts specific to its use. Ipads, understood as technologies that are not clearly bounded by ‘affordances’ 
but participate in various ways in educational activities, contribute to school development and the 
transformation of learning, as described above. Ipads for instance participate in transformative teaching and 
learning processes in the sense that they become part of the ecologies of learning constituted by teachers, 
learners and schools. In the cases described above the iPad becomes part of the dynamics of classrooms in 
which many kinds of resources are used, for instance whiteboards, paper, books, pens, jotters and laptops. As 
a flexible technology, the iPad allows pupils to construct their own systems of related resources or resource 
installations that suit their particular and shifting learning needs. In this sense the iPad contributes to 
inclusive uses of technologies and educational resources that may enhance inclusive educational settings. 
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ABSTRACT 

Over the past years, a number of international initiatives that recognize the importance of sharing and reusing digital 
educational resources among educational communities through the use of Learning Object Repositories (LORs) have 
emerged. Typically, these initiatives focus on collecting digital educational resources that are offered by their creators for 
open access and potential reuse. Nevertheless, most of the existing LORs are designed more as digital repositories, rather 
than as systems for organizing and sharing educational communities’ explicit knowledge (depicted in digital educational 
resources constructed by teachers and/or instructional designers) and tacit knowledge (depicted in teachers’ and students’ 
experiences and interactions of using digital educational resources available in LORs). Within this context, in this paper 
we present an approach for designing LORs as systems for facilitating the organization and sharing of different types of 
educational communities’ knowledge. 


KEYWORDS 

Learning Object Repositories, Educational Communities, Knowledge for educational practice. Knowledge of educational 
practice. Knowledge Management 


1. INTRODUCTION AND PROBLEM DEFINITION 

Today it is commonly argued that, digital educational resources generated by teachers and by students, as 
well as by teacher-to-students and students-to-students interactions during day-to-day educational activities 
constitute core knowledge assets of educational communities (Chen, et al, 2009; Hsu et al, 2008; Carroll et 
al, 2005). Within different educational communities (Wenger and McDermott, 2002), digital educational 
resources are worthy to be organized, managed, shared and reused effectively (Hsu et al, 2008). For this 
purpose, a number of international initiatives have emerged recently and they have recognized the 
importance of sharing and reusing digital educational resources among educational communities typically 
represented in the form of Learning Objects (LOs) (McGreal, 2004; UNESCO, 2002). 

Most of the above mentioned initiatives provide systems and services that aim to support the web -based 
management of LOs. A particular category of those systems is the Learning Object Repositories (LORs), 
which are developed to facilitate search, retrieval and access to LOs (Geser, 2007). McGreal (2008) has 
defined LORs as systems that: “enable users to locate, evaluate and manage learning objects through the use 
of “metadata”, namely descriptors or tags that systematically describe many aspects of a given learning 
object, from its technical to its pedagogical characteristics”. Nevertheless, most of the existing LORs are 
designed as digital repositories of educational resources providing functionalities only for the organization 
and sharing of educational communities’ explicit knowledge (typically depicted in digital educational 
resources constructed by teachers and/or instructional designers), whereas functionalities for the organization 
and sharing of educational communities’ tacit knowledge (typically depicted in teachers’ and students’ 
experiences and interactions using digital educational resources available in LORs) are very limited. 
However, both aforementioned knowledge types are very important to be managed, shared and reused 
effectively among educational community members (McLaughlin and Talbert, 2006). 
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This is actually a major drawback in the design and development of LORs, since there is not a common 
approach for designing LORs towards addressing the problem of organizing and sharing different types of 
educational communities’ knowledge. 

In previous work, reported in (Kallonis and Sampson, 2010) an initial study of existing LORs from the 
Knowledge Management Systems (KMSs) perspective has been performed and a list of essential LORs’ 
functionalities has been proposed that could address the problem of organizing and sharing educational 
communities’ knowledge. In this paper, we adopt these functionalities and we present an approach for 
designing LORs towards addressing the current demands of web -facilitated educational communities, as well 
as the generation, sharing and application of different types of knowledge among educational community 
members. 


2. KNOWLEDGE MANAGEMENT IN WEB-EACILITATED 
EDUCATIONAL COMMUNITIES OE PRACTICE 

Communities of practice have become increasingly influential within several fields since they are identified 
as an important mechanism through which individual and group knowledge is created and transferred (Cox, 
2005). The concept of communities of practice has been proposed by Lave and Wenger (1991), who define a 
Community of Practice (CoP) as: “a group of people who share an interest, a craft, and/or a profession. It can 
evolve naturally because of the member's common interest in a particular domain or area, or it can be created 
specifically with the objective of gaining knowledge related to their area of interest”. CoPs that are facilitated 
by web-technologies are referred to as web -facilitated communities of practice or virtual communities of 
practice (Hara et al, 2009). 

The concept of CoP has also become very popular in the field of education and learning. As a result, 
educational communities of practice are being developed focusing on generating, sharing and reusing 
different types of educational knowledge [9]. The different types of educational knowledge, which can be 
generated and shared within educational communities of practice, can be divided into two types (Cochran- 
Smith and Lytle, 1999): 

■ Knowledge for educational practice: this is formal knowledge depicted in the LOs that are 
constructed by teachers and/or instructional designers of an educational community and they can be 
used to enhance teachers’ day-to-day educational practice. This type of knowledge can be considered 
as explicit, since it can be articulated codified and stored in certain media (Ronald and Kulkarni, 
2007; Tiwana, 2003). 

■ Knowledge of educational practice: this type of knowledge is constructed: (a) by teachers based on 
their experiences about their students’ learning and evidenee of their progress in relation to given 
LOs, (b) by students based on their experiences about the use of given LOs provided by their 
teachers, and (c) by teachers-students interactions with these LOs. This type of knowledge can be 
considered as tacit, since it needs special effort to be codified and transferred (Tiwana, 2003). 

In order to build systems that facilitate the aforementioned knowledge types in the context of web- 
facilitated educational communities, Charlier et al (2007) and Goel et al. (2009) have identified a set of needs 
for web -facilitated communities of practice that should be addressed by these systems. For the purpose of our 
work, we have compiled and adapted them accordingly, so as to be applicable to web-facilitated educational 
communities of practice and they are presented below as requirements of such systems and in relation with 
the aforementioned knowledge types: 

■ Requirement 1 - Stimulate participation and interaction: That is, to stimulate the participation of 
the educational community members and to foster their active involvement in creation and sharing 
their own LOs towards constructing explicit knowledge for educational practice, as well as in 
evaluation and tagging of LOs towards constructing tacit knowledge of educational practice. 

■ Requirement 2 - Accommodate informal and spontaneous interactions: That is, to promote and 
foster communication and interactions that are not intended and planned but happen by chance 
towards constructing tacit knowledge of educational practice. 
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■ Requirement 3 - Empower the individual: That is, to empower the individuals in web -facilitated 
educational communities, so as to facilitate them in the process of sharing explicit knowledge for 
educational practice depicted in their own LOs and tacit knowledge of educational practice depicted 
in their experiences in using available LOs. 

■ Requirement 4 - Foster and stabilize commnnity members’ relationships: That is, to facilitate 
the creation of stable relationships between community members, so as to raise the level of 
collaboration, communication and contribution within the community by constructing and sharing 
either explicit knowledge for or tacit knowledge of educational practice. 

■ Requirement 5 - Build trust between community members: That is, to facilitate the creation of 
trusted relationships between community members, so as the explicit knowledge for and the tacit 
knowledge of educational practice that is shared to be considered as valid and useful. 

■ Requirement 6 - Simplify access to the community: That is, to provide the appropriate facilities 
(infrastructure, tools and services) that would simplify access to the community, so as to enable 
sharing and transferring of explicit knowledge for and tacit knowledge of educational practice. 

■ Requirement 7 - Distinguish different levels of participation: That is, to identify active 
contributing members of the community and acknowledge them as the core contributing group 
regarding the explicit knowledge for and the tacit knowledge of educational practice, which are 
constructed and shared among community members. 

■ Requirement 8 - Maintain the sense of being part of the community: That is, to provide the 
appropriate facilities (infrastructure, tools and services) that would raise the members’ sense of being 
part of the community towards enabling them to actively participate to the construction and sharing 
of explicit knowledge for educational practice and tacit knowledge of educational practice. 

Additionally, in order to support typical knowledge management processes in the context of web- 
facilitated communities of practice, Tang et al. (2009) have identified eight (8) specific activities that web- 
facilitated community members should perform. For the purpose of our work, we have adapted these 
activities accordingly, so as to be applicable to web -facilitated educational communities of practice and they 
are presented below in relation with the aforementioned requirements: 

■ Activity A - Construct Knowledge: During this activity the members of the community (either as 
individuals or as members of a group) create new LOs (that is explicit knowledge for educational 
practice) and/or they provide their experiences in using available LOs (that is tacit knowledge of 
educational practice) using the available infrastructure. Both educational knowledge types can then 
be shared within the community (Activity C - Share Knowledge). 

■ Activity B - Synthesize Knowledge: During this activity the members of the community (either as 
individuals or as members of a group) use the existing educational knowledge in its explicit form 
(namely, LOs) and/or in its tacit form (namely, experiences in using available LOs via forum 
discussions, blog posts, social tagging, personal messages and/or wikis), in order to support Activity 
A - Construct Knowledge. 

■ Activity C - Share Knowledge: This activity is twofold. The members of the community (either as 
individuals or as members of a group) (i) share the explicit educational knowledge (LOs) that was 
constructed during Activity A and/or (ii) share their tacit educational knowledge through web 2.0 
tools (namely, blogs, wikis, social tagging and social networks) 

■ Activity D - Learn: During this activity the members of the community (either as individuals or as 
members of a group) use the knowledge presented in the community by either searching/retrieving it 
(Activity H - Search/Retrieve Knowledge) or by using Web 2.0 tools (Activity B - Synthesize 
Knowledge), so as to enhance their learning. 

■ Activity E - Evaluate Knowledge: During this activity the members of the community (either as 
individuals or as members of a group), perform some type of formal or informal (through simple 
reflections) evaluations on the educational knowledge which is presented in the web -facilitated 
educational community. The members may rate and comment on the appropriateness of the LOs 
presented in the community by using Web 2.0 tools (Activity B - Synthesize Knowledge). 

■ Activity F - Distill Knowledge: During this activity the members of the community (either as 
individuals or as members of a group), assess the design of explicit educational knowledge (depicted 
in LOs), in order to identify patterns that may lead to the extraction of general designs for later use 
and/or reuse. 
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■ Activity G - Apply Knowledge: During this activity the members of the community (either as 
individuals or as members of a group) use the educational knowledge which is available in the 
community by applying it in their own educational practices. This can lead to the creation of new 
explicit and/or tacit educational knowledge (Activity A - Construct Knowledge). 

■ Activity H - Search/Retrieve Knowledge: During this activity the members of the community 
(either as individuals or as members of a group) search and retrieve the existing educational 
knowledge that is available within the community, in order to support all the above mentioned 
activities. 


3. LEARNING OBJECT REPOSITORIES AS KNOWLEDGE 
MANAGEMENT SYSTEMS 

In previous work, reported in (Kallonis and Sampson, 2010) an initial study of existing LORs from the 
Knowledge Management Systems (KMSs) perspective has been performed and a list of essential LORs’ 
functionalities has been proposed that could address the problem of organizing and sharing educational 
communities’ knowledge. Table 1 presents an extended list of LORs functionalities from a knowledge 
management perspective. 

Table 1. Extended List of LORs’ Functionalities from a Knowledge Management Perspective 

Description 

LOs Component 

This functionality enables LORs’ end users to store in the LOR their LOs and/or links 
to external LOs, so as to be able to reference them with unique URLs for future use 
and sharing them with other users. 

This functionality enables LORs’ end users to search LOs using appropriate 
commonly agreed terms which are matched with metadata descriptions of the LOs 
This functionality enables LORs’ end users to browse LOs according to different 
classifications based on their metadata descriptions 
This functionality enables LORs’ end users to preview the content of the LOs 
This functionality enables LORs’ end users to download the LOs and further use them 
or modify them locally (when the license associated with this LO permits 
modifications) 

This functionality enables LORs’ end users to provide their ratings and comments for 
the LOs stored in a LOR. 

This functionality enables LORs’ end users to bookmark LOs and add them to their 
personal and/or favorite lists, so as to be able to access them more easily in the future 
This functionality analyzes users’ previous actions regarding LOs search and 
retrieval, and it automatically recommends to them appropriate LOs that are related 
with the LOs that has been previously searched and retrieved 
This functionality is used in order to provide LORs’ end users with better rankings of 
LOs during their searching, which are based on other users’ comments and ratings 
Mash-ups refer to web applications which present data acquired from different 
sources and combined in a way which delivers new functions or insights. This 
functionality enables LORs’ end-users to perform fedearted searches and retrieve LOs 

from other LORs. 

Metadata Component 

This functionality enables LORs’ end users to store in the LOR the metadata 
descriptions of their LOs, so as to be able to reference them with unique URLs for 

future 

This functionality enables LORs’ end users to view in details the metadata 
descriptions of LOs, so as to be able to decide whether to use or not a specific LO 
This functionality enables LORs’ end users to download the metadata descriptions of 
LOs in XML format conformant with IEEE LOM Standard, so as to further process 
them with appropriate educational metadata authoring tools and upload them back to 
the same LOR or to another LOR 

This functionality is used for validating the appropriateness and the quality of the 
metadata descriptions provided for the LOs by their authors and in many LORs this 
functionality is available to a limited number of back-end users (namely, metadata 
experts), who undertake the task to ensure the quality of metadata descriptions 
This functionality enables LORs’ end users to characterize LOs by adding tags to 
them. 


LORs Functionalities 

Store 

Search 

Browse 

View 

Download 

Rate/Comment 

Bookmark 

Automatic Recommendations 
Knowledge Filter 

Mash-ups 

Store 

View 

Download 

Validate 
Social Tagging 
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Pesonal Accounts 


Forums 
Wikis 
RSS Feeds 

Blogs 

Social Networks 


Other Added‘Value Services 

This functionality enables LORs’ end users to create and manage their own personal 
accounts by completing their personal information and preferences. User accounts 
include also information about: (a) the LOs that a user has contributed to the LOR, (b) 
the LOs that the user has bookmarked and (c) the ratings/comments and tags that the 
user has provided to the different LOs of a LOR 

This functionality enables users to communicate and exchange ideas in an 
asynchronous way about the use of LOs that are stored in a LOR 
This functionality facilitates users to create wikis and share information about their 
experiences with the LOs that are stored in a LOR 

This functionality enables users to be informed via RSS readers about new LOs, 
which are added to the LOR without visiting the LOR 

This functionality enables LORs’ end-users to build and maintain their own blogs for 
publishing their opinions about LOs stored in LORs and receiving comments from 
other end-users about their reflections 

This functionality enables LORs’ end-users to build online social networks based on 
the LOs that they are offering to the LORs, so as to share their common interests. 


As we can notice from Table 1, there are functionalities related to three (3) different components which 

constitute a LOR, which are: 

• Learning Objects Component Dimension: The functionalities related to this LOR component enable 
LORs’ users to interact with either the LOs locally hosted by the LOR or the links to externally hosted 
LOs, in various ways, such as store, search, browse, view, download, rate/comment, bookmark and 
automatic LOs recommendations 

• Learning Objects Metadata Descriptions Component Dimension: The functionalities related to this 
LOR component enable LORs’ users to interact with the metadata descriptions of the LOs, in various 
ways, such as store, view, download, validate and social tagging. 

• Added-Value Services Component Dimension: The functionalities related to this LOR component aim 
to enhance the experience of the LORs’ users in relation to the other two dimensions including services 
such as the creation of personal accounts, forums, wikis and RSS feeds for new LOs added to the LOR. 
Next, we consider these functionalities and we present an approach for designing LORs as systems for 

managing educational communities’ knowledge. 


4. AN APPROACH FOR DESIGNING LORS AS SYSTEMS EOR 
MANAGING EDUCATIONAL COMMUNITIES’ KNOWLEDGE 

Porter (2008) has proposed the Activities, Objects and Features (AOF) method for designing web portals 
which includes three (3) general steps. For the purpose of our work, we have adapted these steps, so as to be 
applicable for designing LORs that aims to address the requirements of web -facilitated educational 
communities and support knowledge management activities of educational communities’ members: 

■ Step 1 - Focus on the primary Activity: This step includes the identification of the primary activity 
that the users perform in a web portal. For the case of LORs the primary activity is the organization 
and sharing of the different types of educational knowledge (namely, explicit knowledge for 
educational practice and tacit knowledge of educational practice) 

■ Step 2 - Identify the social objects: This step includes the identification of the objects that users 
interact with while performing the primary activity defined in step 1 . For the case of LORs, the social 
objects are the LOs that are available to the educational community members of these LORs. 

■ Step 3 - Choose the core feature set: The final step includes the identification of the core feature set 
that will facilitate the users of the web portal to perform actions on the social objects defined in step 
2. For the case of LORs, the core feature set could be based on the extended list of LORs 
functionalities (as presented in section 3). The specific functionalities for designing a LOR could be 
selected according to the requirements of web -facilitated educational communities (as presented in 
section 2) that need to be addressed and the generic knowledge management tasks (as presented in 
section 2) that need to be performed by educational community members within the designed LOR. 
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To this end, for each requirement and KM activity identified in section 2, we can identify those 
functionalities that are needed to meet each of these requirements and KM activities. Table 2 presents 
the mapping between the extended list of LORs’ functionalities and the requirements of web- 
facilitated educational communities addressed by each functionality, whereas Table 3 presents the 
mapping between the extended list of LORs’ functionalities and the KM Activities that are supported 
by each functionality. 

Table 2. Mapping Extended List of LORs’ Functionalities to Requirements of Web-Facilitated Educational Communities 


Requirements of Web-Facilitated Educational Communities 


LORs Functionalities 

R1 R2 R3 

R4 

R5 

R6 

R7 

R8 

Store 

LOs Component 

✓ 



✓ 


Search 

✓ 

- 

- 

- 

- 

- 

Browse 

✓ 

- 

- 

- 

- 

- 

View 

✓ 

- 

- 

- 

- 

- 

Download 

✓ 

- 

- 

- 

- 

- 

Rate/Comment 

✓ 

- 

- 

- 

✓ 

- 

Bookmark 

✓ 

- 

- 

- 

- 

- 

Automatic Recommendations 

✓ 

- 

- 

- 

- 

- 

Knowledge Filter 

✓ 

- 

- 

- 

- 

- 

Mash-ups 

✓ 

- 

- 

- 

- 

- 

Store 

Metadata Component 
✓ - - - 





View 

✓ 

- 

- 

- 

- 

- 

Download 

✓ 

- 

- 

- 

- 

- 

Validate 

✓ 

- 

- 

- 

✓ 

- 

Social Tagging 

✓ 

- 

- 

- 

✓ 

- 

Pesonal Accounts 

Other Added-Value Services 
✓ ✓ 

✓ 

✓ 

✓ 

✓ 

Forums 

✓ ✓ 

✓ 

✓ 

- 

- 

•/ 

Wikis 

✓ ✓ 

✓ 

✓ 

- 

- 

✓ 

RSS Feeds 

✓ 

- 

- 

✓ 

- 


Blogs 

✓ V 

✓ 

✓ 

- 

- 

✓ 

Social Networks 

✓ ✓ 

✓ 

✓ 

✓ 

- 

✓ 
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LORs Functionalities 

Knowledge Management Activities of Web-Facilitated 
Educational Communities 

A B 

C 

D 

E E 

G 

H 

Validate 

✓ 

- 

- 

✓ 

- 

- 

Social Tagging 

✓ ✓ 


- 

✓ 

- 

- 


Other Added-Value Services 



Pesonal Accounts 

- 

- 

- 

✓ 

- 

- 

Forums 

✓ 

✓ 

✓ 

✓ 


- 

Wikis 

✓ 

✓ 

V' 

✓ 


- 

RSS Feeds 

- 

- 

- 

- 

- 

- 

Blogs 

✓ 

✓ 

V' 

✓ 


- 

Social Networks 

✓ 

✓ 

V 

✓ 


- 


As we can notice from Table 2, it appears that all requirements of the web -facilitated educational 
communities of practice can be addressed by the extended list of LORs’ functionalities. Moreover, as we can 
notice from Table 3, it appears that all KM Activities executed by the members the web -facilitated 
educational communities of practice can be addressed by the extended list of LORs’ functionalities. As a 
result, designing LORs that follows the proposed approach can support the management of the 
web -facilitated educational communities’ explicit and tacit knowledge. 


5. CONCLUSIONS AND FUTURE WORK 

In this paper, it was argued that it is important to design LORs that address the problem of organizing and 
sharing educational communities’ explicit knowledge (depicted in digital educational resources constructed 
by teachers and/or instructional designers) and tacit knowledge (depicted in teachers’ and students’ 
experiences and interactions of using digital educational resources available in LORs). For this purpose, we 
propose an approach for designing LORs that address the requirements of web -facilitated educational 
communities and support knowledge management activities of educational communities’ members. 

Future work includes the exploitation of the results of this work for evaluating whether the features of 
existing LORs are addressing the tasks that need to be performed by educational community members for 
organizing and sharing the different types of educational communities’ knowledge. 
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THE CONFIGURATION PROCESS OF A COMMUNITY OF 
PRACTICE IN THE COLLECTIVE TEXT EDITOR 
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ABSTRACT 

The various tools available on Web 2.0 enable the interactions in a Community of Practice (CoP) to be optimized and 
may discourage the participation of members. Thus, the choice of the tools is fundamental for the growth and 
maintenance of a CoP. With a focus on this and, from the analysis of the characteristics of the group and the activities 
which will be developed, this article deals with the configuration process of a Community of Practice in the CTE - 
Collective Text Editor . As such, the article discusses the CoP and presents the CTE as the meeting place and use of this 
community, since it corresponds to the demands and needs of the group. 


KEYWORDS 

Community of Practice, Collective Text Editor. 


1. INTRODUCTION 

The term "community" may bring to mind something that is increasingly less experienced in large urban 
centers: the sense of community as a meeting, in person, of people affectively united by something in 
common. For Burbules (2004), it is a nostalgic memory of the community, as it relates to the "memory of a 
time when affiliation was based on proximity, on relative homogeneity and familiarity: the community of a 
small town, a neighborhood, a large family " (p. 209). 

Nowadays the term community is most frequently used together with other nouns or adjectives (school 
community, virtual community etc.) and reflects what is understood, in different spheres, as the best strategy 
for achieving results (efficient, effective, etc.): cooperativeness. In this sense, whether to increase the 
productivity of businesses or to enhance learning processes, among other reasons, people seek to meet (or are 
reunited) in order to form communities. 

One type of community that has emerged from these necessities are the Communities of Practice (CoP). 
These communities generally unite people interested in specific learning and in the practical application of 
this (Terra, 2005). 

When the objective is uniting people, the Internet and Web 2.0 show one of their greatest vocations: 
interaction. Communities of Practice take strength from the opportunities of connection anytime, anywhere; 
of collective construction and, often, of the lack of expense offered by free software, making the virtual 
world your space par excellence. 

The idea of configuring a CoP arose based on the needs of a group of postgraduate students from the 
Federal University of Rio Grande do Sul (UFRGS), to meet virtually, to give continuity to the discussions 
realized in person and to share and produce new texts and study topics. Thus, the focus of this paper is to 
relate the configuration process of this Community in the Collective Text Editor (CTE), pointing out the 
reasons which show why this environment is suitable for this community. It is hoped that this study will 
present a text editor as a possible space for the establishment of a Community of Practice. 

As such, the first section of this article discusses the Communities of Practice. The second presents the 
CTE. The third section presents the reasons which pointed to the Editor as the option that best meets the 
needs, values, knowledge and expertise of the members of the CoP. Lastly, the concluding remarks are 
presented. 
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2. COMMUNITY OF PRACTICE 

According to Wenger (2006), "Communities of Practice are groups of people who share an interest or a 
passion for something which they learn to better by interacting regularly." As for Terra (2005, p.Ol), "CoPs 
consist of people who are connected, informally as well as contextually, by a common interest in learning, 
and principally in practical application." 

Wenger (2006) points out that the Communities of Practice differ from other communities due to 3 major 
characteristics: 

a) Domain: the theme or topic of interest to the community, in which the members of the CoP feel 
compromised. 

b) Community: formed through the relationships between members and allows individuals to learn from 
each other. 

c) Practice: A CoP is not just a community of interests, it is the set of shared resources (experiences, 
stories, tools and mode of referring to current problems, among other things) that form the repertoire for the 
use of everyone in the resolution of problems. 

According to Schlemmer (2012), when the project upon which the Community of Practice is working 
ends, the CoP also ends. Wenger (1998, as cited in Ribeiro,Silva, Boff and Viccari, 2011) and Terra (2005) 
defend, from this standpoint, that CoPs have cycles or stages of life. 

According to Wenger (1998, as cited in Ribeiro et al, 2011), these cycles are called creation, expansion, 
maturation, activity and dispersal. As for Terra (2005), cycles are birth, growth, maturity, decline and death. 
However, although the CoP has "a well-defined life cycle, in relation to its stages, [...] there are no limits on 
the temporal scope for the definition of each of the stages" (Ribeiro et al, 2011, p. 696 ). 

In this sense, for a Community of Practice to remain active during its period of maturation or maturity, 
Wenger, McDermott and Snyder (2002) propose 7 principles of management: 

1. Planning for evolution: thinking that in the future the CoP may have new and different needs; 

2. Maintaining the dialogue between the internal and external perspective: opening possibilities for other 
exchanges, encouraging them among members and among other people and communities; 

3. Inviting to different levels of participation: understanding that people are different, and therefore, 
interact in different ways and degrees; 

4. Developing public and private spaces (one-on-one) for the community members; 

5. Focusing on the value of the CoP: The communities survive because, as well as their members, they 
are valued. As the authors point out, the value is important because, in the majority of communities, adhesion 
and permanence are free. 

6. Combining familiarity and stimulation: Familiarity with tools and activities is great for members to feel 
comfortable in the CoP, but it is also necessary to offer new things and to encourage participation; 

7. Creating rhythm for the community: maintaining regular events and avoiding overloading. 

In view of the principles mentioned by Wenger, McDermott and Snyder (2002), one realizes that the life 
cycle of a Community of Practice is strictly related to its members and, naturally, the interactions between 
them. It is in this sense that Wenger, White, Smith and Rowe (2005) point out how it is necessary and 
fundamental to know the members as well as the activities they perform to choose the most appropriate tools 
for the CoP. After all, as the authors affirm, when the user does not learn the technologies with ease, he may 
feel discouraged to participate. Thus, the tools can play a role in optimizing the interactions and giving 
support to collective work or, conversely, may discourage the participation of members as well as harming 
the collective work. 

For Wenger et al (2005) a perfect technological configuration does not exist. The most appropriate 
configurations will always vary from community to community. Thus, one should take into account the level 
of access to and command over the technologies that participants have; the capacity to connect; the browser 
used; the availability of purchase or the preference for free software; the need (or not) for technical support, 
etc.. 
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Besides these aspects, the type of activity that the community realizes and which the technology must 
mediate, must be analyzed. Examples are; interaction activities, publishing, sharing, etc. From this 
understanding, it is possible to choose the tools focused on the needs (and possibilities) of the members of the 
community. 

The Web 2.0 tools have long been used by the CoP. As pointed out by Kirkwood (2006), Web 2.0 allows 
people with a particular interest in common to find other people with the same interest and form their 
communities. 

It is noteworthy also that, besides providing tools for interaction, Web 2.0 encouraged the development of 
Free Software, that is, free online tools. This, in Brazilian terms, may be fundamental to the existence of 
CoPs focused on learning outside educational platforms such as Moodle (http://www.moodle.org.br/). 

It should be noted also that the free tools of Web 2.0 have become very popular and are part of everyday 
life for many people. For communities of practice focused on learning, using these tools is to give students a 
sense of familiarity. By providing this feeling, students are free to make relationships, encouraging them to 
create, share, publish and cooperate - fundamental principles of pedagogical theories based on the 
transmission of knowledge, that go beyond traditional teaching. 

Thus, tools like blogs and wikis, which allow both collective and individual creation and publication, can 
be highlighted; synchronous tools like msn and skype, podcasts, which enable the exercise of creativity, 
move away from textual activities, etc. It is possible to find all these resources, as well as forum, polls and 
others, free on the web. Using them or not in a CoP will depend on factors related to the command and 
characteristics of the participants. 

Given the above, the next section presents the Collective Text Editor (CTE), an environment of collective 
construction of texts, coupled with synchronous and asynchronous tools, free and available on the web. 


3. THE COLLECTIVE TEXT EDITOR - CTE 

The Collective Text Editor - CTE (http://www.nuted.ufrgs.br/etc2/index.php) - was developed by 
NUTED/UFRGS (Center for Digital Technology applied to Education at the Federal University of Rio 
Grande do Sul - http://www.nuted.ufrgs.br/) aiming initially to promote collective writing. The editor is 
online, and therefore does not need installation in a server or local computer. 

Fully developed within the philosophy of Free Software, the CTE employs PHP language and uses the 
related database management system (DBMS) MySQL and the Apache Web server, both with open source 
code. It also has customer focused technologies, such as JavaScript, Dynamic HTML and Cookies, among 
others (Macedo, Zank, Vinade and Behar, 2010). 

In 2009, the CTE was completely restructured, becoming reliant upon a new interface, new interaction 
features and a new logo. Moreover, files are now organized following a folder structure, as shown in 
Figure 1: 



tii documentos 
CV biblioteca 
forum 

r dados pessoais 
Q ajuda 
£ sair 


Meus Documentos - CdP da EPT 


[ Nova pasta ] [ Novo texto 4,^ ] [ Propriedades Q ] [ Configurar ^ 


Exduir ^ ^ Enviar mensagens ^ J [ Estatisticas ^ ] 

Autores ▼ Modificado em 


Q Meus Documentos 
^ EDU3027 
Competenci... 
Discutindo ... 

Q EPT e a Com... 
Q Oficinas Vi... 

^ SEG -Turma... 
Q Equipe ETC 
Q CdP da EPT 


0Ensino Medio Integrado Claudia 


02.12.2011 as 
15:57:36 


Figure 1 . CTE Home Screen. 
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The text can be viewed and edited entirely and can still be written by different users simultaneously, 
which relies on different tools of interaction and communication, among these: a) Message: allows 
participants to send messages via e-mail to each other, b) Forum: has search and editing features, as well as 
different viewing options c) Comments: tool located on text editing screen, allows users to leave messages 
for each other or make observations about the writing d) Communicator: tool that displays online users, 
allowing them to converse in real-time and simultaneously to the editing. 

Other highlights of the CTE are the new interface, which follows usability criteria that make it very 
intuitive, and the tools "Concepts Network" and "MineraForum", which are based on the technology of Text 
Mining. The first allows the extraction and relation of the principal terms discussed in the textual 
productions. Meanwhile, the MineraForum extracts and relates the principal terms discussed in a forum, as 
well as attributing a relevance value for each message posted. 


4. THE CHOICE OE THE COLLECTIVE TEXT EDITOR 

Based on the concept of Wenger (2006) surrounding the Community of Practice, this article relates the 
reasons for which the CTE was chosen as the meeting work and group discussion environment. 

The community is composed of a teacher and five students of the PPGEDU / UFRGS (Postgraduate 
Program in Education at the Federal University of Rio Grande do Sul), which have studies and research 
related to Professional Education in common. The CoP is provisionally titled, the "Community of Practice of 
Professional Education and Technology" or CoP PET. 

According to the characteristics of CoP proposed by Wenger (2006), it can be said that this group already 
has a domain of interest. Professional Education, but that it is still forming as a community. What they lack, 
entirely, is the characteristic of practice. In this sense, even if the community is not yet established in a virtual 
space, it is possible to consider that it already exists, since students meet in person. Based on Wenger (1998, 
as cited in Ribeiro et al, 2011) and Terra (2005), it appears, therefore, that the community is moving towards 
a phase of expansion/growth. 

In order for this CoP to be constituted entirely and arrive in the phases of activity (Wenger, 1998, as cited 
in Ribeiro et al, 2011) and maturity (TERRA, 2005), it was decided to observe the principles laid down by 
Wenger, Mcdermott and Snyder (2002) for the management of CoPs in addition to the suggestions of 
Wenger et al (2005). Thus, to choose the environment and tools, it was necessary to obtain some data with 
community members. Accordingly, a questionnaire was created to infer information about: the degree of 
command over the Information and Communication technologies and the Internet and the values and 
activities that the group intends to perform (WENGER et al, 2005; WENGER, 2006). In relation to the 
values, it was sought to identify whether the CoP members would agree to pay for the technological 
resources and use environments and/or tools available only in English. 

The questionnaire was answered by 6 people, the teacher and his 5 advisees. The collected data was 
compared with the characteristics of the Collective Text Editor, a free online virtual environment developed 
within the University in order to verify that the CTE could be a space option for community practice. This 
comparison showed that the CTE will meet the needs, values, knowledge and mastery of the members of the 
CoP regarding Information and Communication Technologies (ICT), as shown in Table 1: 
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Table 1. Correspondence between the necessities, values, knowledge and command of the Community of Practice 

members in relation to the TIC and CTE. 


Necessities, values and knowledge of 
members of the CoP 

Collective Text Editor - CTE 

Effective use of ICT, Internet and 
social networks: 

a) Don't use nor master many 
technological resources. The majority 
of members (5 people) use only e-mail. 
Half (3 people) have an account in a 
social network, although only one uses 
it frequently. 

- Is available online (download not 
necessary); 

- Has an interface familiar to other 
editors, which facilitates its use and 
makes learning how to use it fast. Has a 
friendly interface and is intuitive. 

Command of the English Language: 

a) The 6 members stated that they had 
some knowledge of the English 
language but prefer to use tools and 
environments available in Portuguese. 

Available in Portuguese. 

Payment: 

a) Don't wish to pay for the virtual 
space. 

- It is a Free Software and therefore is 
free of cost. 

Activities most realized: 

a) Discussions, publication and sharing 

Possesses synchronized and 

unsynchronized tools which take 

account of these activities and needs. 


Beyond the domain of the community and also of the needs, values and knowledge of the members, the 
possible styles that the CoP covers must be taken into account in order to choose the environment or the tools 
that will be used. The styles relate to the group of activities that are performed by members of the community 
and that should result in a set of tools to support these activities. 

Based on Wenger, White, Smith (2009), it is understood that this community covers "meeting", "open 
conversations" and "content" activities. The functions, tools that can give support to these activities and 
corresponding tools in the CTE are presented below; 

a) Meeting - Strong activity and feature of the community. However, part of the regular scheduled 
meetings will continue to be conducted in person (face to face). Thus, the CoP will be mixed, since there will 
be face to face and virtual encounters. 


Table 2. Activities and Tools - Style “Meeting”. 


Activities which 

contribute to 

characterizing the style 
“Meeting” 

Tools which would give 
support to these 

activities 

Corresponding tool (s) 
available in the CTE 

Booking 

Shared calendar; E-mail; 
Utility software for 

booking 

Message 

Synchronized Interactions 

Video-conference: Web- 

conference and 

webcasting; 

Teleconference and VoIP; 
Chat Rooms 

Communicator 

Unsynchronized 

Interactions 

Discussion forum; Wikis; 
E-mail lists 

Forum; message 

Presence/attendance 

Attendance tools; Folders; 
Photos of the participants 

Communicator; Personal 
Details 

Participation in and 

taking of decisions 

Poll 

Without corresponding 

tool. 


b) Open Discussions - The conversations between members remain permanently open, being extremely 
important to everyone’s learning. 
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Table 3. Activities and Tools - Style “Open Conversations”. 


Activities which 

contribute to 

characterising the style 
“Open Conversations”: 

Tools which would give 
support to these 

activities 

Corresponding tool (s) 
available in the CTE 

Conversations about a 
topic at a time 

Email; Email lists; Chat; 
Eunctionality of blog 
comments; etc. 

Message; Communicator 

Conversations about 

multiple Topics 

Eorums on the Web; 
Wikis; Discussion trails in 
blogs; categories; 

aggregation services; 

Microblogging. 

Forum - Collective Text 
Editing 

Sub-groups/privacy 

Access control 

Report mechanisms for 

the wider group 

Access control for the 
texts (definition of the 
participants) and message 
for the participants of the 
folder 

Highlighting key learning 
points -> utilization of 
features which highlight 
the most recent/active 
collective discussions and 
constructions 

FAQs; Wikis for 

summaries; Tags; 

categories; Evaluation 

mechanisms of the posts; 
Tools which highlight the 
active discussions. 

Forum; Editing of 

collective text; wall with 
new posts (Forum), new 
contributions in the 

collective texts and new 
messages. 

Filing 

Web repositories for email 
lists; Automatic filing for 
the Forum; Permanent 
links in blogs; Tag Clouds 

Automatic filing of the 
forums; history of 

messages sent and 

received; history of the 
versions of the collective 
texts. 


c) Content - The greatest interest of the CoP with respect to the environment will be in the opportunity to 
share and give/have access to documents, tools, and diverse content. It is thought that the possibility for 
collective writing motivates the production of scientific articles in the group. 


Table 4. Activities and Tools - Style “Content”. 


Activities which 

contribute to 

characterising the style 
“Content” 

Tools that would give 
support to these 

activities 

Corresponding tool (s) 
available in the CTE 

Sharing of document files 

Independent document 

repositories; discussion 

annexes 

Library 

Comments, notes and 
content discussions 

Discussion forums; Wikis 
for notes; Blogs with 
comment features; Web 
page noting tools. 

Forum, collective text 
editing and comments 

Publication of Content of 
one's production 

File sharing; Blogs; Web 
pages; Wikis. 

Collective text editing, 
text url, and library (also 
as a portfolio) 

Distributed editorial 

capacities 

Tagging; Evaluation; 

Comments 

Comments 

Filing 

News with time control; 
Automatic filing. 

Automatic filing of the 
forums; history of 

messages sent and 

received; history of the 
versions of the collective 
texts. 
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Given the above, it was understood that the CTE could correspond to the needs, values and knowledge of 
the members and could also give support to the activities of the community. It shows itself, therefore, as an 
appropriate space for the CoP to form completely and reach the phases of activity (Wenger, 1998, as cited in 
Ribeiro et al, 201 1) and maturity (TERRA, 2005). 


5. FINAL THOUGHTS 

This article describes the configuration process of a Community of Practice (CoP) in the Collective Text 
Editor (ETC). To this end, the first section sought to present what the Communities of Practice are and how 
they are configured. Next, the Editor was presented. In sequence, the needs and opportunities of the members 
of the CoP regarding technological resources as well as the activities to be performed and the tools that can 
give support to these activities were dealt with, indicating the corresponding tools and activities in the CTE. 

The Community of Practice in question is formed by a group of postgraduate students. These students 
already meet in person in order to discuss issues relating to Professional Education. Based on Wenger (2006), 
it is understood, therefore, that this group, in addition to already possessing knowledge, is now also forming 
itself into a community. 

The CTE is a virtual environment that is freely available on the web. The Editor was initially developed 
with the aim of promoting collective writing. However, also relying on interaction and communication tools, 
both synchronous and asynchronous, the editor can serve as a meeting and work space of a Community of 
Practice. 

The data analysis collected in questionnaires, which were answered by community members, point to the 
Collective Text Editor (CTE) as an appropriate environment for the needs of the group. Likewise, the Editor 
can respond to the needs and difficulties of the members regarding foreign languages and command of digital 
resources. It is further added that the CTE corresponds to the styles that the CoP covers, that is, it relies on 
tools that can give support to the activities groups "Meetings", "Open Conversations" and "Content". 

Eor these reasons, the CoP was implanted in the CTE and is already active. The interactions are occurring 
frequently, primarily through the Eorum and Message tools. In light of this, data is being collected. The 
analysis of this data can then validate the Editor as a space for the formation of Communities of Practice. 
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ABSTRACT 

Increasingly, faculty academics are required to teach and design online courses. However, in many cases, faculty 
members report having low levels of confidence, self-efficacy and competence to teach in online environments. Although 
their professional learning is often enhanced by institutional support strategies such as workshops, online instruction and 
mentoring systems, many faculty academics learn through “just-in-time” rather than “just-in-case” strategies. 

This paper reports on the findings from a cross-continental research project between researchers in two higher education 
institutions in the United States and Australia. The project was initiated to: 1) determine the learning needs of faculty 
members who teach online and design online courses; and 2) to develop tailored professional learning programs and 
resources to enable faculty members to become effective online teachers and skilled online course designers. 

As well as providing an account of the research findings to date, the paper provides recommendations for other 
researchers who may be considering cross-institutional or cross-continental research about online teaching, online course 
design and professional learning programs. 


KEYWORDS 

Online teaching; professional learning; self-efficacy; threshold concepts; research collaboration 


1. INTRODUCTION 

1.1 Background and Significance 

The body of literature examining online instruction sheds light on the challenges faced in many higher 
education institutions. Zhen, Garthwait, and Pratt (2008) studied factors influencing 400 faculty members’ 
decisions to employ multiple forms of online course management applications (such as Blackboard and 
WebCT). The researchers found that motivational factors such as personal philosophies and self-efficacy had 
a strong impact on the probability of adopting online course management applications. The lack of clearly 
articulated pedagogy for online instruction presents further challenges to online teachers. Shephard, Madelon, 
Alpert, and Koeller (2007) argue that instructors’ presumed lack of technical competence, knowledge, and 
anxiety contribute to inefficacious beliefs for teaching online. The authors highlight the need for continued 
and varied professional development opportunities to bolster the efficacy of online educators. 

Teachers’ beliefs in their self-efficacy affect their general orientation toward the educational process as 
well as how they design instructional activities (Pajares, 1992; Prosser & Trigwell, 1997; Samuelowicz & 
Bain, 2001; Trigwell & Prosser, 1996). Teachers with a strong sense of self-efficacy are more inclined to 
demonstrate greater levels of planning, organization, eagerness, and allot additional time teaching in areas 
where their sense of self-efficacy is higher, while tending to steer clear of subjects and topics when self- 
efficacy beliefs are lower (Tschannen-Moran & Woolfolk Hoy, 2001). They also tend to be more accepting 
of new ideas, more eager to try new techniques, are more dedicated to teaching, persevere during tumultuous 
endeavors, recover better in light of problems, are less critical of students who make mistakes, and work 
longer with students experiencing difficulty (Gibson & Dembo, 1984). Furthermore, their experiences of 
teaching challenges and “bumpy moments” can catalyze their professional learning (Romano, 2006). These 
findings suggest that fostering teacher self-efficacy is integral to sound educational practices. 
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In light of the challenges surrounding online education, the primary aim of the research is to examine 
online teaching self-efficacy, threshold concepts and troublesome knowledge that are encountered by faculty 
as they develop their online teaching skills. The second aim of the research is to develop a set of research- 
informed guidelines, pedagogies, and practices that will enhance the development of professional training 
programs for online education. Threshold concepts are defined as “core concepts that once understood, 
transform perception of a given subject” (Meyer & Land, 2003a) and troublesome knowledge (Perkins, 2006) 
is knowledge that can cause cognitive conflict as learners compare new ideas with their prior knowledge. The 
literature in this area has detailed several challenges of online education including developing one’s identity 
in a virtual environment and overcoming technology to better facilitate instructional practice (Carmichael, 
2012; Koole, 2010; Savin-Baden, Sinclair, Sanders, & Wind, 2007). 

1.2 Conceptual Framework 

Bandura’s Social Cognitive Theory provides the theoretical lens from which to examine the dimensions 
pertaining to online teaching self-efficacy beliefs. Research has established the close association between 
teacher efficacy and commitment to teaching, adoption of innovations, and use of effective strategies 
(Albion, 2001; Bandura, 1997; Kulinna & Silverman, 2000). Because self-efficacy beliefs are conceptualized 
to be multidimensional and are specific to a particular area or domain, the examination of self-efficacy 
beliefs within online education will extend the theory of self-efficacy to this instructional medium. 

1.3 Specific Aims 

The initial aim of this investigation is to examine the dimensions and perceptions of postsecondary online 
instructors through interdisciplinary and transcultural collaboration. Through this process, the research seeks 
to identify the challenges faculty face in online course development and instruction. The long-term goal of 
the project is to provide empirically-based evidence to inform the customised development of future 
professional learning activities. Through interdepartmental, institutional, and cultural comparisons, we also 
aim to delineate the contextual, or cross-continental, factors that influence teaching and learning. These 
research questions outline the study’s aims: 

1 ) What are the threshold concepts that faculty encounter when they learn about online education? 

2 ) What are the online teaching self-efficacy beliefs of faculty, and do these beliefs converge or depart from 

their identified threshold concepts? 

3) Are differences in online teaching self-efficacy and threshold concepts seen across disciplines and 

academic institutions? 

4 ) How can the identification of online teaching self-efficacy and threshold concepts using diverse samples 

of postsecondary online educators be used to inform future academic development programs? 


2. METHODOLOGY 

Drawing from the work of Northcote and her colleagues, this research seeks to build upon an established 
framework of online teaching threshold concepts and self-efficacy evaluation developed by the study’s 
researchers (Northcote, Reynaud, Beamish, Martin, & Gosselin, 2011). As part of this collaborative effort, 
two of the study’s researchers (Northcote and Gosselin) met in Cooranbong, Australia at Avondale College 
of Higher Education in October of 2012 to administer, collect and analyze the study data. 

2.1 Research Design 

A mixed-methods triangulation design was used for this research (Creswell & Plano Clark, 2007). The design 
involves simultaneous collection and analysis of qualitative and quantitative data throughout the project. The 
study consisted of three steps. The first two steps were conducted concurrently at both researchers’ affiliated 
institutions. The final step integrated these data and provided the cross-continental comparisons of online 
teaching beliefs. 
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2.1.1 Step 1: Reflective Journals and Focus Groups 

The researchers in the study gathered data in the form of systematic reflective journals and focus groups 
approximately two times per month across a four month period. These data provided information about the 
general concerns and breakthroughs reported by online instructors, and were evaluated to identify threshold 
concepts related to online teaching. Findings from these data analyses were examined to determine the 
relatedness to the framework established through earlier research. 

2.1.2 Step 2: Questionnaire and Analysis of Results 

The Online Teaching Self-Efficacy Inventory (Gosselin, 2009) was administered at the time of the final round 
of reflective journaling. Data from this questionnaire provided a means of measuring the degree of 
self-efficacy about faculty members’ online teaching abilities and provided comparative data with the 
qualitative findings from Step 1 . 

2.1.3 Step 3: Formulation of Guidelines for Future Professional Development Activities 

Findings from analyses of data from the reflective journals and focus groups (Step 1) and the questionnaire 
instrument (Step 2) were contrasted with the initial work of Northcote et al. (201 1) to holistically evaluate the 
articulated dimensions on online instruction. Similarities and differences were gleaned from these data by 
comparisons across data collection sites. Finally, these steps informed the development of best practices for 
future professional learning programs in online instruction which augment the programs and resources that 
have been provided to date (for example, Northcote, 2011; Northcote, Reynaud, & Beamish, 2011; 
Northcote, Seddon, & Brown, 201 1). 

2.2 Instruments 

The OTSEl: Online Teaching Self-Efficacy Inventory (Gosselin, 2009) is a five-scale inventory consisting of 
46 total items to assess multiple components of online teaching self-efficacy. Online faculty participants were 
asked to indicate how confident they are in accomplishing the activities by selecting a number for each item 
on a scale ranging from 0 to 10. No confidence is represented by a 0 and complete confidence is indicated by 
a 10. The inventory scales include: (1) Web-Based Course Structure', (2) Online Curricular Alignment, (3) 
Course Content Migration', (4) Virtual Interaction', and (5) Selection of Technological Resources. Alpha 
reliabilities of the scales range from .84 to .95 reflecting excellent internal consistency. The average variance 
accounted for across the five single-factor scales ranges from 45.93% to 64.38% with an average of 53.16% 
of explained variance. The percentage of variance explained for each of the inventory scales provides 
evidence for good factor validity (Stevens, 1996). 

The reflective journal provided the researchers with a format to record their observations of how faculty 
members encountered both successes and challenges in their online teaching and course design processes in a 
typical semester. The reflective journal template incorporated three points of reflection; the first two being 
prompter-questions and the third being a list of key issues, about online teaching; 

• Reflection 1: Erom my point of view, what are the major concerns or areas of “troublesome 
knowledge” that faculty talk to me about or that 1 observe? 

• Reflection 2: What typical questions do faculty members ask me or others about online learning? 

• Reflection 3: Do the faculty ask about or comment on the following concepts? (for example: the 
nature of the online learning environment; student attention; online communication) 

The focus groups aimed to extrapolate additional challenges, concerns, and perspectives surrounding 
online teaching and course development. The semi-structured format consisted of nine questions to examine 
the following: 1) areas of concern in online education; 2) perceptions of students’ concerns in the online 
environment; 3) the nature of online pedagogy in contrast to face-to-face instruction; and 4) experiences 
interacting with colleagues, support staff, and administration in developing courses and teaching online. 
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2.3 Sample 

The population for the study includes faculty currently teaching and/or developing online courses at The 
University of Texas at Tyler (UTT) and at Avondale College of Higher Education (ACHE) in New South 
Wales, Australia. Eor the purposes of this investigation, online courses must have 50% or more of the content 
delivered through online mediums. Twenty participants from each site were solicited for participation in the 
qualitative aspects of this research. The accessible population for the quantitative phase includes 311 (273 
from UTT and 38 from ACHE) faculty currently teaching undergraduate or graduate courses online. A total 
of A = 121 (38.91%) completed and returned the OTSEI for inclusion in the analysis of data. 

2.4 Data Analysis 

Content analysis was conducted to evaluate the reflective journals and focus group responses (see Step 1 
above). Descriptive data from the Online Teaching Self-Efficacy Inventory (OTSEI) (see Step 2 above) was 
analyzed and contrasted with the qualitative findings. Additionally, data across partnering institutions and 
departments have been compared for potential differences and similarities. Outcomes of these data analyses 
were used to inform the planning process for future professional learning programs. 


3. FINDINGS TO DATE 

3.1 Avondale College of Higher Education 

Preliminary analyses of the data gathered in the reflective journals which were kept by researchers at 
Avondale College of Higher Education have been conducted. The data recorded the concerns expressed and 
the typical questions asked by faculty members about online teaching and online course design. Eindings 
from these analyses indicate that, overall, faculty are mainly concerned about learners and the quality of 
learning. When describing how faculty members were putting their ideas about online education into 
practice, words such as “tension”, “suspicion”, “doubt” and “experimentation” were mentioned to describe 
the transition that many faculty members were experiencing between teaching in on-campus contexts to 
online environments. Because Moodle is used as the primary learning management system (LMS), comments 
about understanding how Moodle works is also a focus of many of their comments. 

As part of their data gathering methods, the researchers in this study kept records of the questions that 
faculty members asked about online teaching, online course design and online learning. Many of these 
questions were characterised by references to online enrolment systems, technical functions of the LMS, 
interactive and communication facilities of the LMS and uploading materials to the online courses. 

The major concerns related to areas of “troublesome knowledge” (Perkins, 2006) and threshold concepts 
(Meyer & Land, 2003b) that faculty members reported to the researchers of this study reflected a tension 
between knowing what was possible in online teaching and having the pedagogical and technical skills to 
apply these ideas to practice. Eaculty members made requests for increased professional learning 
opportunities and more streamlined enrolment and support systems. 

3.2 University of Texas at Tyler 

The preliminary results from the University of Texas at Tyler sample resulted in the articulation of challenges 
in five main areas including: 1) technology; 2) design; 3) time; 4) student interaction; and 5) support. The 
comments made by faculty were eclectic in nature and included: a desire to emulate face-to-face interaction 
and to connect with students online; challenges with incorporation of new technology; difficulty reaching 
students that may prefer to remain distant; the need to go beyond expectations to make the content interesting 
to engage students in the online environment; and that online education is much more time intensive on the 
both the development and teaching than traditional, face-to-face courses. In regard to support, the instructors 
expressed that while they felt comfortable should they be required to develop courses without support, having 
instructional design and technological support staff was needed to have a “...refined, polished, and pretty” 
layout for the courses. 


285 



ISBN: 978-989-8533-18-0 © 2013 lADIS 


Support staff mentioned that adoption and integration of new technology was a primary challenge for 
faculty as well as students. Additionally, the support staff perceived that both faculty and students 
underestimated the amount of time it takes to learn new technology. This caused an exorbitant amount of 
pressure from both faculty and students at the onset of each semester due to requests for assistance in a 
concentrated time period. 

Both faculty and support staff expressed that asynchronous interaction allowed for time to think before 
responding, and the nature of these discussions were perceived as richer than what might be seen in face-to- 
face classes. Both groups also felt that the administration was not aware of the vast amount of time required 
to develop and teach online courses. The disconnect was thought to have come about due to a general 
misrepresentation of what online teaching entailed; the participants expressed concern that those 
administrators having little to no experience in online education were unaware of the time, effort, and 
challenges it takes to develop quality online courses and educational experiences. 

3.3 Online Teaching Self-Efficacy 

Table 1 provides the means and standard deviations across the UTT (n = 67) and ACHE (n = 54) samples on 
the five OTSEI scales. On average, the UTT faculty participants rated themselves as least efficacious on the 
Web-Based Course Structure scale (M = 6.71, SD = 1.54) while the ACHE faculty group reported the lowest 
self-efficacy for the Selection of Technological Resources scale (M = 4.00, SD = 2.13). The Selection of 
Technological Resources Scale examines online teachers’ self-efficacy in their ability to select, utilize and 
determine the appropriateness of technology to enhance student learning and enrich instruction. The Web 
Based Unit Structure scale assesses self-efficacy beliefs that comprise the ability to construct and design 
online units that include clear organizational structure, facilitates straightforward navigation and 
communication guidelines, and is aligned with an institution’s mission. Both of these scales involve the 
construction and design of online courses, and interestingly, both groups were the least efficacious in 
carrying out these aspects of online education. Conversely, both the UTT group (M = 7.65, SD = 1.05) and 
ACHE group (M = 5.92, SD = 2.26) reported the highest self-efficacy appraisals on the Online Curricular 
Alignment scale. The scale encompasses faculty’s self-efficacy beliefs in their ability to effectively align 
learning objectives, course assignments, assessment strategies, and learning activities within online courses. 

Table 1. Means, Standard Deviations, and Comparisons across Data Sites for the OTSEI Scales 


Selection of Technological Resources 

M 

SD 

f(119) 

Cohen ’s d 

95% Cl 

ACHE" 

4.00 

2.13 

8.39* 

1.53 

[1.12, 1.94] 

UTT*’ 

6.80 

1.54 




Virtual Interaction 






ACHE 

5.44 

2.16 

6.06* 

1.11 

[0.73, 1.50] 

UTT 

7.32 

1.19 




Course Content Migration 






ACHE 

5.28 

1.95 

7.51* 

1.37 

[0.98, 1.77] 

UTT 

7.48 

1.25 




Online Curricular Alignment 






ACHE 

5.92 

2.26 

5.57* 

0.14 

[-0.22, 0.49] 

UTT 

7.65 

1.05 




Web Based Course Structure 






ACHE 

5.33 

2.09 

4.20* 

0.10 

[-0.26, 0.46] 

UTT 

6.71 

1.54 





n = 54; \ = bl\*p< .001 
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Comparisons were made using independent measures t-tests across the five OTSEI scales for each faculty 
group. The results of the independent samples t-test indicated significant differences across all OTSEI 
subscales {p < .001) with the UTT group reporting higher self-efficacy beliefs across all of the inventory 
scales. Additionally, the UTT group had taught significantly more semesters online (M = 5.37, SD = 5.13) 
compared with the ACHE group (M = 3.00, SD = 3.49). The results of an independent measures t-test 
demonstrated a significance difference in the number of semesters taught online by each group, f(l 19) = 2.88, 
p = .005. Although the UTT sample had spent more time teaching online, they reported less total years 
teaching in higher education (M = 8.04, SD = 5.29) than the ACHE group (M = 11.97, SD = 9.36). The 
results of the independent t-test to assess the differences in total years teaching in higher education was 
significant, f(l 19) = 4.20, p < .001 . 

At this point in the research, comparisons of the combined OTSEI data show significant differences. 
Specifically, across all scales, the sample from UTT reported higher levels of online teaching self-efficacy. 
When evaluating the number of years that each sample had taught online, the UTT sample demonstrated 
significantly more experience in online instruction. According to Bandura’s theoretical conception for the 
formation of self-efficacy beliefs, the primary development of one’s self-efficacy beliefs is through enactive 
mastery experiences (Bandura, 1997). One possibility for the differences in groups in that the UTT sample 
had spent more time teaching online, and as such, we would expect them to have developed higher self- 
efficacy beliefs through proportionally more experience in online instruction. 

3.4 Recommendations for Online Training Programs 

The findings of the research so far, including outcomes of the analyses of data from questionnaires, reflective 
journals and focus groups, have been considered holistically to inform the future professional learning 
activities that are planned for each institution. The evidence from this research indicates that faculty members 
require professional learning opportunities that enable them to extend their pedagogical and technical skills 
in online teaching and course design. 

To meet the needs of the faculty members who participated in this study, the following professional 
learning activities are currently being planned and implemented: 

• Online and printed resource development to provide faculty with self-paced instructional 
materials to extend their online teaching and online course design skills. An example would include the 
development of an online tutorial for instructors that outlines the process for using advanced features of 
lecture recording software. 

• On-campus and online workshops to guide faculty in their development of online teaching and 
course design skills such as workshops focused on the design of learning activities and assessment. 

• Templates and examples of online courses are being developed and sourced to demonstrate 
effective examples, protocols and formats used in effective online courses and online teaching methods. 
Seasoned online educators are currently developing templates that newer instructors may use in guiding 
the development of their own courses. 

• Online education showcases to feature online courses being taught by faculty within each 
institution and across institutions. A rotating system of cross-departmental workshops using online 
educational delivery systems is being considered to allow for exposure of varied and eclectic 
instructional approaches. 

• Research about online teaching to continue to provide an evidence-based approach to inform the 
development of future professional learning programs. With the dynamic and shifting nature of online 
teaching technologies and delivery systems, it is recommended that ongoing, cyclical evaluation be 
carried out within departments and institutions to promote evidence based support to facilitate best- 
practices in online education. 

• Direct involvement of administrators with faculty and staff to inform the allocation of resources 
and support in the successful development and teaching of online courses and programs. It is 
recommended that department chairs, deans, provosts, and other administrators be included in online 
teaching workshops and program evaluation to demonstrate the need for support of online educators. 
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4. CONCLUSIONS 

The findings of this research to date indicate the close relationship between the extent of experience that 
faculty members have teaching in the online environment and their self-efficacy levels. Although differences 
in online teaching self-efficacy were seen across institutions, analysis of the qualitative data showed that each 
group faced similar areas of concerns for teaching and developing courses in the online environment. This 
research has also provided evidence of the close beliefs-practice link between teachers’ ideas about teaching 
and their practical approaches to teaching within the context of online education. 

This investigation employs a mixed methods design triangulation design, and consequently gives a voice 
and contextual understanding of the challenges related to online education within the broader theoretical 
framework of social cognitive theory. By examining both the contextual factors and self-efficacy beliefs of 
online educators, a holistic understanding of the challenges related to online teaching is uncovered that better 
informs the development of tailored professional training programs compared to using singular approaches. 

The findings that emerged from this research study indicate that the concerns of the faculty members who 
participated in this study were primarily related to learning and learners. The faculty members also showed a 
“big picture” understanding of where online education was placed within their institution’s operations and, as 
such, expressed concern that the systems that supported online education (for example, enrolment, IT, student 
services) should support and not hinder the development and implementation of online courses. 

The benefits of the cross-continental research collaboration that has formed the basis of this project were 
many and varied. Lessons learned in each institution were shared, as were resources and research 
instruments. Although separated by vast geographical distances, the challenges and successes of the faculty 
members in both institutions are remarkably similar; both sets of faculty members were concerned about 
pedagogical issues and technical issues, and they both expressed a need for institutional systems so support 
online education. The main difference between the groups of faculty in each institution was their length of 
experience in online teaching and, consequently, their online teaching efficacy levels. Although this 
investigation found many similarities across instructional faculties within institutions in Australia and the 
United States, future investigations are needed to delineate the nature of this relationship across additional 
educational systems that are varied by country and associated institutional culture. The researchers 
recommend replication of the methodology used in this investigation as a cost-effective and comprehensive 
approach from which future investigations may be based. Through this process, the external validity of the 
results may be generalized to other populations. 

The research instruments used in this study, the OTSEI: Online Teaching Self-Efficacy Inventory, the 
focus group questions and self-reflection journals, were relevant to both institutions and faculty members 
across a variety of disciplines. Only minor adjustments were required across the two countries to ensure the 
language used was relevant to local faculty members. Sharing these instruments increased the efficiency of 
the research timeframe and overall processes. These instruments could be used by researchers in other 
institutions to pursue similar investigations about online teaching. 

The findings from this research have reinforced the need for professional learning programs to be tailored 
to suit faculty members’ needs, rather than offering professional development activities that might work or 
that faculty members might need. Instead, the researchers have found that “just-in-time” professional learning 
programs are more appreciated and accessed by faculty than those offered on a “just-in-case” basis. 
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ABSTRACT 

Each semester faculty members at a regional university encountered students in their courses who were unprepared for 
learning. As the demand for rigor continues to increase in all fields of study continues to increase, professors expressed 
concern regarding preparedness of their students to enter the work-force. In an effort to leverage the learning taking place 
under their direct supervision, faculty members re-designed their courses and placed responsibility for basic content 
acquisition on the students. When students arrived in class each week, students were actively engaged in the learning 
process through student-centered learning events. After class, students were required to demonstrate reflective 
practitioner skills in a real-world format. Application of digital tools allowed students to learn the basic content and 
demonstrate learning in a manner conducive to their learner preferences. Students were engaged in the learning process 
before, during and after class. The course redesign instructional model provides a framework for CELDA participants as 
they address main issues with the evolving learning processes and technical applications in the digital age. 
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1. INTRODUCTION 

With growing frequency, faculty members in a regional state university expressed frustration when university 
students did not read assigned materials and would not disengage from digital connectivity in their 
classroom. In a like manner, the university students noted, in the course evaluations, technology was used in 
varying degrees throughout the learning process. The inconsistency with which faculty members utilized the 
standard course management system (Blackboard) also frustrated students, specifically, the organizational 
structure of the courses (Institutional Research, 2012). In order to address the disillusionment of students and 
the frustration of faculty members, the university invited faculty members to volunteer and re-design their 
courses based on research and best practice in teaching today’s young adults. Research on contemporary 
challenges facing the country was examined and the effective techniques of adult education explored. The 
effective application of technology in the teaching and learning process became the linchpin for course 
re-design. 


2. REVIEW OF LITERATURE 
2.1 Contemporary Challenges 

Results of the 2010 Programme for International Student Assessment (PISA) released in December of 2010 
rekindled the United States’ frustration with the state of public school instruction. While Shanghai, Korea, 
Hong Kong, Singapore, Finland, Canada, Japan and New Zealand ranked in the top seven, the United States 
ranked 25th on an internationally standardized assessment of 15-year-old math students (Guria, 2010). PISA 
rankings take into account differences among culture and economic systems. As a result, PISA provides 
benchmarks to measure existing differences while providing countries flexibility in setting realistic goals that 
meet national conditions. 
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As global competition intensifies and productivity patterns change, the United States will need creative 
reform of attitude and strategies to maintain an economic role in global markets. Furthermore, because 
countries with limited resources consistently outperformed United States’ students, increase in money alone 
is clearly not the solution. 

In addition to increasing educational competition, the United States faces growing international 
competition for the job market. Pink (2006) suggested material abundance has shifted social emphasis from 
survival to a greater emphasis on “self’ development in terms of beauty, spirituality and emotions. In 
contrast, Asian countries have focused more directly on economic development. Graduating huge numbers of 
degreed workers willing to work for less income, Asia has rapidly absorbed outsourced jobs. Simultaneously, 
United States white-collar jobs are lost through rapid improvements in automation and online technology. 

Economic sustainability is reliant upon a competitive work force. Education remains the best tool for 
creating future global workers; however, changes in public education are necessary to reflect marketable, 
international skills. The challenge of remaking a relevant educational system is complicated when one 
considers the evolving need to effectively address a most “different” type of learner. 

2.2 Needs of Learners 

While some debate the existence of generational characteristics (Bennett & Maton, 2010; Helsper & Eynon, 
2009; Trzeniewski & Donnellan, 2010), many contend that contemporary students think, behave, and learn 
differently due to ubiquitous exposure to technology (Prensky, 2001; Tapscott, 2009; Taylor 2005). 
Contemporary learners equipped with hyper connected and multi-tasking digital brains, are unprepared to 
endure the slow pace of instructional practices developed more than a century ago (Sprenger, 2009). 
Consumer and entertainment oriented, intellectually disengaged in non-digital environments (Taylor, 2005). 

Physical evidence corroborates brain differences resulting from exposure to digital media in terms of how 
digital learners process, interact, and apply information (Juke, 2006). Stimulation and adaptability enables the 
brain to constantly reorganize or rewire itself (Jensen, 2005; Willis, 2008). Brains of digital learners are 
physically different from those of learners who have not experienced ongoing exposure to technology. 
Digital learners fundamentally think and process information differently than their predecessors by using 
multi-tasking and parallel processing; they prefer graphics to text, random access (hyperlinks), networking, 
instant gratification, frequent rewards, and games rather than “serious” work (Prensky, 2001). Jensen (2005) 
recommends instruction based on problem solving, critical thinking, relevant projects, and complex activities 
that stimulate the brain and challenge learners. Interactive feedback must be specific, timely, and learner- 
controlled while addressing multiple modalities. 

Combining digital learner processing skills and learning preferences with brain research further justifies 
breaking from the current teaching/learning paradigm in which professors control content, delivery, and 
products in favor of authentic learner engagement. Carmean and Haefner (2002) suggest deeper learning 
principles are required to help engage digital learners in meaningful content processing. 

2.3 Instructional Engagement 

To evolve from passive content-consumers to active information-processors requires instructional 
engagement. Engaged learners work collaboratively, transforming understanding through creative problem 
solving (Jones, Valdez, Nowakowski, & Rasmussen, 1994). Taylor (2010) noted authentic engagement 
occurs when educators furnish students with enough skills and tools to become self-motivated. Schlechty 
(2001) stresses students learn best in applied learning tasks, emphasizing engagement is an active and 
interactive process, and not synonymous with time on task. Engaged students learn more, retain more, and 
enjoy the learning activities more than unengaged students (Dowson & Mclnerney, 2001; Hancock & Betts, 
2002; Lumsden, 1994; Yoke, 2002). Instructional goals that create opportunities for authentic engagement, 
where students meet expectations and intended instructional outcomes responsive to learner interests and 
values, produce the most effective learning (Schlechty, 2002). 
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2.4 Instructional Technology 

Through interaction and exploration in creative and innovative ways, technology empowers students to 
communicate and socialize beyond the classroom. No longer limited to physical space, an expanded 
classroom can accommodate community -driven, interdisciplinary, and virtual collaboration. This provides an 
unprecedented opportunity for schools to reexamine traditional approaches and current practices, and 
redesign parameters of effective instruction (The Horizon Report, 2010). Supporting educator effectiveness 
by expanding innovative learning models that utilize online and blended learning, high-access, technology- 
enriched learning environments, and personalized learning models will increase student learning (Bennett & 
Maton, 2010). 

More research is needed to understand how to design technology-infused, learner-centered instruction. 
Technology in an educational setting should focus on instructional goals rather than technology innovation. 
Furthermore, deeper understanding is needed regarding the instruction models designed to support 
technology-rich environments. Puentedura (2008) identified four levels of technology use in class instruction: 
substitution, augmentation, modification, and redefinition (SMAR). Created to help teachers reflect and 
refine their use of technology in instruction, the first two levels of the SMAR model focus on instructional 
enhancement, technology as a tool substitute, but provides no functional change (testing on computer instead 
of paper). At the next level, technology still substitutes for a conventional tool, but with functional 
improvement (watching a video versus modeling the process). However, at the transformation level 
technology significantly improves instruction. Technology used to redesign or create original tasks result in 
richer, more engaged and integrated learning at higher levels of thinking. 

2.5 Instructional Innovation 

Instructors with content competence and an arsenal of effective instructional strategies produce higher 
achievement outcomes among students (Coleman, et ak, 1966; Rowan, Correnti, & Miller, 2002; Whitehurst, 
2002). Ensuring transformation in educational processes requires creation of new instructional models by 
effective instructors; however, many university professors are reluctant to embrace this shift due to 
technology availability, cultural lag (change in universities happens more slowly), and instructional training 
(Chen, 2007; Johnson, 1997; Maddux, 1997). Nevertheless, Cavanaugh, Dawson, & Rithaup (2011) found 
infusing integrated instructional development, support, and technology produces significant changes in 
teaching practices: direct instruction decreases, collaboration and project-based learning increases, and 
student motivation and engagement improve. Specific to technology integration, Somekh (2000) found 
educational institutions that recognize the importance of information and communication technology promote 
integrated instruction to transform the learning process more effectively when reform is implemented in a 
bottom-up fashion. 

Teaching digital learners demands different instructional strategies. Educators today must engage digital 
learners and create instructional opportunities by utilizing technology to empower learners. Universities must 
move from automating processes (testing, email) to informing processes that empower students to solve 
problems, access information and create relationships outside the classroom using the tools of technology 
(November, 2010). 


3. COURSE RE-DESIGN PROCESS 
3.1 Rationale 

Eaculty members at a regional university located in the southwestern portion of the United States discussed 
students’ apparent inability to adequately prepare for class participation. Students did not complete reading 
assignments or prepare themselves for intellectual participation in class discussions. Students were not 
satisfied with their learning experience at the university. Through end-of-semester course evaluations, 
students noted the frustration they experienced because faculty members did not use current technology in 
the learning or evaluation process. 
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Courses presented inconsistencies in the use of the university course management system. Confusion and 
frustration were the results of this practice (Institutional Research, 2012). 

To effectively prepare a new generation of people for the country’s work-force, Arthur Levine (2010) 
asserts education programs must transform the use of traditional educational practices such as lecture, note- 
taking and dated textbooks. Today’s students, both in higher education and public schools, are digital natives 
(Pink, 2006). These students routinely use technology for acquisition of new information, communication 
and collaborative projects. As a result, students must be engaged in authentic learning experiences using 
skills from the real-world. Best practice in education suggests students must be actively engaged in the 
learning process for content to become meaningful (Schlechty, 2001). Engaged students retain information 
longer and apply the content in new situations (Jensen, 2005). 

3.2 Faculty Cohorts 

Based on the concerns of professors/students and best practice research, cohorts of faculty members decided 
to transform a set of initial courses across disciplines at the regional university. The faculty members 
developed an alternative content delivery method, held students accountable for the acquisition of the basic 
information before class, and imbed technology in the learning process. As a result, the engaged students 
critically applied new knowledge in an interactive setting: the classroom. Following each class session, 
students reflected on the basic content, the peer-to-peer interactions during class and applied the learning to 
future work -related settings. 

This collaborative effort in course redesign simultaneously embraced students’ digital connectivity and 
enhanced content acquisition. The components of course redesign included: 

1 . Acquisition of basic course content information prior to class attendance; 

2. Evidence of basic content knowledge upon arrival to class; 

3. Actively participate in authentic learning experiences; 

4. After class, students reflected on lessons learned. 

3.3 Re-design Components 

Faculty members developed a wide variety of learning objects to be experienced prior to each class meeting. 
The objects were accessed by students and a form of accountability associated with each. For example, the 
professor developed a visual and auditory object the students would access before class (example: Captivate 
product). The student would complete a viewing guide which accompanied the learning object. This process 
ensured the acquisition of basic course content information prior to class attendance. 

Upon arrival to class, the students produced evidence of their basic content knowledge. The evidence took 
the form of a completed viewing guide, question they wanted answered in class that day or a summary 
statement of the most important information gleaned from the learning object at that time. In some 
classrooms, if the student arrived unprepared for class, the instructor directed the student to remain in the 
hallway and complete the assignment. Only when the evidence of basic knowledge was produced to the 
instructor was the student allowed to enter the classroom. These high expectations of class preparation caused 
the classroom to take on the professional environment of the real-world workplace. Arrive prepared for 
work, or go home and return ready to contribute to the learning community. 

During class, students actively participated in authentic learning experiences which deepened course 
content understanding.The experiences were completed in teams and/or individually. In teams, they solved 
real-world problems such as designing an advertising campaign for a local business. They presented the 
results of their work as a team and local business owners provided feedback for improvement and 
acknowledged the strong points of the presentations (authentic audience). Students individually evaluated 
various procedures for solving a problem in their career fields while applying the basic content gained before 
class. At this point, the instructor provided guidance, facilitating the learning at the most critical point: 
application to the current dilemmas which students will encounter as new graduates. Students realized the 
importance of the content as the information was juxtaposed with contemporary issues in their field of study. 

After class, students reflect on lessons learned before class through the learning objects and during the 
interactive learning experiences in the class. The learning follow-up took the form of extended learning 
assignments which directed students to expand current understanding of information. 
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Interviews of people in the field of study were used to enhance student application of course content. 
Reflective journals were also applied as a way of deepening understanding of course content. Students were 
given higher order thinking skills prompts to guide their reflections. Students were directed to apply the 
course content to the real-world field of study and then analyze the potential impact the application would 
have on the future of education. As a final reflection assignment, students created a classroom intervention 
plan to address unique instructional challenges they observed in the public school classroom during the 
semester. 


4. RESULTS 

Access to student course evaluations was not available at the time of this paper. However, a concise 
comparison of course success was submitted by a participating faculty member. By taking the course success 
rate before course re-design and comparing the success rate of the re-designed course, preliminary 
conclusions were drawn. The classes were taught by the same professor, same location and instructional 
equipment and class size was similar. The difference was the application of the re-designed course. 

The course submitted for evaluation was EDU320: Professional Development One: Understanding 
Learners. The course included an examination of students and teachers in learner-centered schools. Topics 
addressed brain-based learning, cooperative learning, learning styles and strengths of diverse learners and 
formal and informal assessment and learner-centered instruction. A technology lab and documentation of 
field experiences were required. The pre-requisites or co-requisite included PSY 220 or 303 or HS 300 and 
minimum of 60 hours toward certification or degree requirements. The grading scale used for this course was 
90-100% earnings of course credit equaled an A, 89-80% a B, 79-70% earned a C, 69-60% earned a D and 
lower than 60 % was failing (F). If a student withdrew from the course, the student received a W on their 
transcript. 

The total number of students in both semesters was 154, and all students were of junior and senior 
standing. Grade comparison of the spring 2011 semester control group (N=81) and the spring 2012 semester 
experimental group (N=73), indicate the percentage of students earning the letter grade A increased from 
32% spring 2011 to 53% spring 2012. As a result, students earning the letter grade B in spring 2012 
decreased to 36% compared to 42% in spring 2011. Additionally, the students earning C decreased in spring 
2012 to 1% as compared to spring 2011 when 12% of the students earned a C. The increased number of 
students earning A in spring 2012 naturally decreased the number of students earning B and C in spring 
2012. 

The percentage of students earning a D remained the same both semesters. Seven percent of the students 
earned an F in spring 201 1 compared to 3% in spring 2012. The percentage of students withdrawing from the 
course was essentially the same both semesters (5% 201 1 and 6% 2012). 

The instructor of record for the submitted course identified three areas of interest for consideration. The 
first challenge was the reflective process of organizing the content of the course so there was a sequence to 
the learning experiences both outside of the classroom and coordination with the learning experiences of the 
classroom. Next was the time required to create the visual and auditory learning objects. Two new software 
programs were applied (Captivate and Soft Chalk). The third and final challenge involved the interactive 
learning experiences for the classroom. The instructor created the learning experiences that would appeal to 
adult learners while holding them accountable for the application of knowledge gained outside the classroom. 
The experiences included solving real-world problems presented in scenario fashion, ranking effective 
teaching strategies for fictional public school students and creating artifacts in teams where each team 
member was responsible for a unique portion of the final product. 

Initially, the findings suggest the re-designed course assisted students earning the letter grades of B and C 
to increase their final grade by a full letter. Course re-design may be the tool that assists student learning and 
in turn produces a more authentically attuned society. In a like manner, course re-design may be the tool that 
assists faculty members to critically evaluate the essence of the course and adjust assignments to reflect real- 
world solutions in the field of study. 
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5. CONCLUSION 

The course redesign model provided a formal vision of how traditional instructional practices could be 
altered to reduce frustration on the part of faculty members and students. The model articulates a process by 
which faculty members work collaboratively to enhance classroom learning and improve their pedagogy for 
the newest type of learners. 

The course re-design model suggests a variety of effective instructional practices to replace frustrating 
traditional strategies. Student frustration decreases because the redesign provides a common look-and-feel in 
course presentation and expectations. Evidently, the consistency of the redesigned courses significantly 
lowered the drop, failure and withdrawal rates. The course redesign instructional model provides a 
framework for instructors as they discern the future direction for higher education programs. 
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ABSTRACT 

A comprehensive literature review that integrates and synthesizes peer-reviewed research surrounding a problem is a 
prerequisite for making an original contribution to a field and profession. The literature review in a doctoral dissertation 
is a required component, but most of these reviews suffer from bounded, linear thinking, limited scope, and little effort to 
go outside of the existing literature on a topic, either in content or in analysis. The intent of this paper is to explore 
concepts of loop learning, dialogic processes, generativity, and methods of communication to reveal common perceptions 
regarding the review of the literature, widen the process and perspective of selecting, review, and critically evaluating the 
literature, and present models that could improve both the process and the outcome of scholarly research. 
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Literature review, dissertation, loop learning, dialogic 


1. INTRODUCTION 

A comprehensive, fully developed literature review is the foundation and starting point for scholarly 
research, and one of the most important tasks in dissertation development. The ability to critically assess, 
integrate, and synthesize peer-reviewed research in an area of interest is a prerequisite for making an original 
contribution to a field and profession. The objectives of a literature review are to provide a path from prior 
studies to a current study, to integrate knowledge, and to stimulate new ideas. Richardson (2003) developed 
the concept of doctors of education as stewards of their field of study. Richardson argued that the literature 
review of a dissertation should elucidate both the practical and the scholarly significance of investigating a 
significant problem. 

Most literature reviews, however, rarely get out of the box in scope and depth. Our experience in 
mentoring doctoral students and researchers over many years suggests that literature reviews often suffer 
from bounded, linear thinking, limited scope, and little effort to go outside of the existing literature on a 
topic, either in content or in analysis. There is a perception among many doctoral students and other 
researchers that the intent of the literature review is to present an extensive, if not exhaustive, summary of 
prior research that supports the centrality and framing of the problem under investigation. This type of 
investigation is used to justify the chosen conceptual framework, identify a gap in the literature, and conduct 
research to improve a given situation. This approach is largely similar to what Argyris (1994) referred to as 
single-loop learning. However, such an approach runs the risk of generating little new insight into known 
problems because of the limited perspective of the search and analysis. 

Truly innovative literature reviews go well beyond simple summaries of source material. A thorough and 
innovative literature review presumes that knowledge is cumulative, and that the researcher not only stands 
on the shoulders of others in exploring a topic of study, but generates critical and substantive value in the act 
of reviewing. Shulman (1999) defined generativity as building on the scholarship and research of those who 
have come before us, and contended that generativity is “one of the hallmarks of scholarship” (p. 162). 
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What is not clear is how a researcher, particularly a novice researcher, can add his or her voice to other 
conversationalists in their field in order to generate profound and useful knowledge. The intent of this paper 
is to explore concepts of loop learning, dialogic processes, generativity, and methods of communication to 
reveal common perceptions regarding the review of the literature, widen the process and perspective of 
selecting, reviewing, and critically evaluating the literature, and present models that could improve both the 
process and the outcome of scholarly research. 


2. THE STANDARD ANNOTATED BIBLIOGRAPHY: SINGLE-LOOP 
LEARNING 

Single-loop learning assumes that problems and their solutions are close to each other in time and space. 
Argyris (1994) described single-loop learning as making minor fixes or adjustments in the current system or 
paradigm, much the way a thermostat is used to regulate temperature. This type of thinking is in accord with 
viewing the literature review as an extensive annotated bibliography, one that stays within the bounds of 
conventional questions, thinking, and scholarship. The researcher reports sequentially on studies related to 
the topic in a manner such that each annotation restates the main argument of a source, the methods of 
investigation, and the main conclusions. At its best, each section of the review contains multiple perspectives, 
and the reader is privy to knowing how the current study could fill a gap identified in the literature. 

Argyris’ (1994) insight was to conceptualize learning as a system of knowledge acquisition, reflection, 
and application. Within the single-loop process the researcher is seeking to understand what has been done to 
solve problems similar to the one under investigation, and focuses on a specific problem that is derived from 
and grounded in the extant literature and dominant conceptualizations of the problem and similar problems. 
The analysis stays largely or entirely within current ways of thinking about the problem, diagnosis, and 
problem resolution, and the analysis is circular and bounded (Cooper, 1988). This type of linear, first-order, 
thinking may be efficient, even necessary at some level, but is not sufficient to explore why a problem exists, 
the generative forces behind the problem, why differences of perceptions and solutions regarding that 
problem exist, and how going outside the dominant ways of thinking about the problem may generate new 
insights that go beyond existing approaches to resolving the problem. 

A single-loop learning process resides within a bounded knowledge space of conventional thinking 
around a problem or question. Understanding and learning are largely confined within the bounds of this 
space, as is dialogue about the problem. The focus is on problem solving, not on exploration of new 
dimensions of the problem or different ways of framing or viewing the problem. Source material for the 
review consists primarily of well established, commonly used sources relevant to the problem. The review is 
usually backward looking, cycling within a constrained domain of sources and interpretations. A missing 
ingredient in many such literature reviews is a healthy dose of criticism, both of the literature gathered and in 
the scope and depth of the process that resulted from the gathering. 

Given the common nature of training of most doctoral scholars, this single -loop process is not very 
surprising. In an extensive review of doctoral programs, Boote and Biele (2005) found most graduate 
students receive little or no formal training in how to review critically the research literature in their field, 
and many students believe their opinion is of little value. The authors contended that it is difficult to perform 
significant research without a profound insight into the field, driven in part by the ability to apply complex, 
higher ordered thinking skills in evaluating prior studies. What is necessary to generate insight that goes 
outside the usual bounds or canon in a field is to undertake a generative as well as summative approach in the 
review. A generative approach that uses dialogical processes to not only summarize the existing literature 
but, goes beyond to include critical evaluation of what is missing from the literature, is a first step in moving 
beyond the single-loop process of most literature reviews. 

2.1 Using Single-Loop Learning in the Literature Review 

The following example of a proposed study illustrates application of single-loop learning to the development 
of a literature review. 


298 



lADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2013) 


Caffeine is the most widely consumed psychoactive drug in the United States, but unlike many other 
such drugs it is legal and unregulated. Caffeine is often used by students studying for exams or when 
they need to stay alert in an early morning class. A variety of outcomes have been reported from people 
who consume caffeine (Rogers et al, 2010; Canales, 2010). However, there is a lack of clarity regarding 
the efficacy of caffeine intake while conducting scholarly research. 

A case study of male and female doctoral candidates who have completed their course work and are 
engaged in dissertation writing will be used to investigate whether or not caffeine consumption is related 
to effectiveness in scholarly research. Participants will keep a daily journal that will include information 
on their daily consumption of caffeine and the research completed each day. An analysis will be 
conducted to determine the relationship between the amounts of caffeine consumed and the quality and 
quantity of their research. 

A first step in conducting the literature review is to create a literature or mind map depicting key 
words related to the variables, problem statement, purpose, research questions, hypotheses, conceptual 
theoretical framework, population, and research method (Simon & Goes, 2013). The literature review for 
this proposed study revealed two possible theories that could guide the research; Arousal Theory and 
Transactional Stimulation Theory (Myers, 2004). A key word search used to find current peer-reviewed 
journal articles included: caffeine as a drug, caffeine and performance; effects from caffeine; benefits of 
caffeine; physical, cognitive, and emotional effects of caffeine; gender and caffeine. 

There are several important limitations to this sort of review. First, the review is dominated by references 
to other studies regarding caffeine as the stimulant. Reliance on a single search referent fits within the linear 
approach of the review, but ignores other potential stimulants that may have similar effects. There is also a 
disregard for potential demographic, cultural, or physiological co-variates that may enhance or detract the 
effects of caffeine consumption, and boundary conditions for the study. For example, the student population 
may adopt caffeine as the stimulant of choice because of ease of access, cultural learning, and dominant 
theories in use about its effectiveness. Similarly, the researcher conducting the review may gravitate towards 
caffeine as the best focus for further research simply because it has received the most attention, and may 
intentionally or unintentionally suppress conflict (Argyris, Putnam & McLain Smith, 1985) over whether to 
widen the scope of the research to consider other stimulants. The theoretical frameworks selected for the 
study may be dominated by those that most closely fit the researcher’s theories in use, even if prior research 
has provided little support for the theory, and thereby foreclose consideration of other theoretical models to 
explain the phenomenon of interest. 


3. DEBATE AND DOUBLE-LOOP LEARNING 

Unlike the reciprocal logic of the single-loop manner of learning, double-loop learning goes outside of 
bounded thinking and established search routines, questioning these routines at a second order level. That is, 
rather than being restricted inside the single-loop system of literature reviewing, double-loop learning 
engages the researcher in a critique of the system, challenging assumptions within the system and 
conventional ways of thinking about a problem to generate new insights and new directions for logic and 
research. Double-loop learning requires holding in abeyance the first and most obvious solutions to a 
problem, while continuing to look for alternative ways of seeing the situation that could lead to a deeper 
understanding. A literature review that includes double-loop learning thereby results in the presentation of 
views both supporting and opposing the researcher’s worldview. 

Such an approach involves a desire to understand what and how some solutions work better than others to 
resolve a problem or achieve a goal. The researcher engaged in this type of thinking views the literature 
review as a debate and dialogue between different ideas and different authors. Gilovich (1991) observed that 
this type of knowing usually entails an intense scrutiny of the views that differ from the researcher’s views, 
but that same critical scrutiny is rarely applied to the studies that support the researcher’s contentions. Much 
like a traditional notion of debate, one side wins and another side loses, while both sides retain their 
certainties. Therefore, at its best, application of double-loop learning develops a deepened understanding of 
assumptions that lead to differing outcomes. 
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In double-loop learning the researcher moves outside of the single-loop process by questioning whether 
dominant theories-in-use and prevailing variables and methods are limiting the scope of inquiry and array of 
source material in the literature review. For example, new variables or covariates might be investigated and 
discussed in the review as a way of identifying gaps or weaknesses not only in the dominant, canonical 
literature around a problem, but also in the broader literature. Rather than limiting the review to common 
theoretical models and search parameters, the researcher widens the review to look at potential covariates that 
appear logically connected to the core focus of the study, but contradicts conventional wisdom. Similarly, 
where existing theories are found to be inadequate or unsupported in investigating a question or issue, 
theories and models connected by logic or context may be brought into the discussion. Continuous 
questioning of assumptions and existing literature, and framing and reframing of the problem, are both means 
to widen the inquiry and step outside of the conventional frame. 

Forays outside the constrained single-loop review may widen the array of variables in the model, or 
identify promising literature sources that are beyond the usual domain of the subject. These new perspectives 
are brought back into the literature review model to widen the inquiry, address the inadequacies of the single - 
loop approach, and provide a more robust literature review that is more likely to generate unique insights, 
different perspectives on the problem, new directions in research problems and methodology, and a more 
comprehensive review. 

3.1 Using Double-Loop Learning in the Literature Review 

The following example illustrates application of double-loop learning in the development of a literature 
review: 

Caffeine is the most widely consumed psychoactive drug in the United States, but unlike many other 
such drugs it is legal and unregulated. Caffeine is often used by students studying for exams or when 
they need to stay alert in an early morning class. A variety of outcomes have been reported from people 
who consume caffeine (Rogers et al, 2010; Canales, 2010). However, there is a lack of clarity regarding 
the efficacy of caffeine intake while conducting scholarly research. 

A case study of male and female doctoral candidates who have completed their course work and are 
engaged in dissertation writing will be used to investigate whether or not caffeine consumption is related 
to effectiveness in scholarly research. Participants will keep a daily journal that will include information 
on their daily consumption of caffeine and the research completed each day. An analysis will be 
conducted to determine the relationship between the amounts of caffeine consumed and the quality and 
quantity of their research. 

A key word search would contain the same terms used in the single-loop review but also include a search 
for opposing views such as: negative effects of caffeine; caffeine as a depressant; health risks of caffeine 
use; etc. As Gilovich (1991) noted, opposing views usually attract greater scrutiny and flaws and are 
more likely to be noted to help build the case for the researeher’s view. Nonetheless, this type of 
reflection will provide a deeper means of defending and justifying a position. 


4. TRIPLE-LOOP LEARNING: DIALOGUE AND COLLECTIVE 
UNDERSTANDING 

Good research is good because it advances collective understanding. To achieve this, in addition to 
understanding what others have done before and explicating the strengths and weaknesses of existing studies, 
the scholarly researcher seeks to deepen understanding of why a complex problem or situation exists and 
what research is necessary to help resolve all or a component of the problem. Unlike the double-loop model, 
the triple-loop approach focuses on a lasting transformation, or a substantive shift in point of view, while 
re-envisioning how to solve problems (Peschl, 2007). Peschl posited that a triple-loop learning model serves 
as a tool to establish context as well as deciphering answers to decisions based on a search for relative truths. 
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Researchers can use triple-loop learning to develop innovative and effective approaches to understanding 
complex issues. This type of learning challenges the researcher to understand the context on how problems 
and solutions are related and how new knowledge is ascertained. 

The results of this learning include enhancing ways to comprehend and change the purpose of the 
research being conducted by developing a better understanding of the situatedness (Lindblom & Ziemke, 
2003) of a problem while deepening the comprehension of why a study was conducted. This portends that 
within a given situation, there is not a single context of relevance but a great number of different, possibly 
overlapping contexts, and that the development of in depth knowledge requires a social and cultural 
embedding. Determining what is a relevant piece of information in a situation is influenced by what Franklin 
(1995) calls the intercontext, since the same situation given in different contexts provides different relevant 
pieces of information. 

Triple-loop learning includes what Bakhtin (1973) referred to as a dialogic expression, which is a 
combination of the individual’s opinions as well as the ideas and thoughts of others. Bakhtin contended that 
an idea is important when it influences others to take action. A literature review based on triple -loop learning 
involves an attempt to: (a) understand how problems and solutions are related, even when separated by time 
and space; (b) understand why previous actions led to current problems; (c) uncover and question premises 
and cognitive patterns; (d) synthesize prior research in ways that permit new perspectives to emerge, and (e) 
establish a rationale for new research. This form of learning can provide a profound understanding of our 
own and others beliefs, perceptions, and understandings. Peschl (2007) proposed that triple-loop learning is a 
type of double-loop thinking about double-loop learning. 

Buber (1996) used the term dialogue to describe a mode of exchange between people in which there is a 
true appreciation of the other, even when their viewpoints are in opposition. Buber expressed the need to 
affirm the person as the bearer of a conviction. When conducting a scholarly literature review, there is an 
opportunity to create a virtual dialogue and respect among authors and policy makers who have attempted, 
within the literature, to resolve problems similar to the one under investigation. The literature reviewer 
becomes a participant in this virtual dialogue by exploring ways of resolving disputes and dead-ends in the 
literature by introducing new directions and rethinking old arguments in light of new research designs that 
could improve discussions in the field. 

Isaacs (1993) defined dialogue “as a sustained, collective inquiry, into the processes, assumptions, and 
certainties that compose everyday experience” (p. 25). Ellinor and Gerard (1998) posited that the dialogue 
process does not hinder disagreement but can foster differing views and create a divergent conversational 
process that could lead to a deeper or new understanding and, in some cases, a natural resolution or 
convergence of thinking. This goes beyond framing the conversation as a debate, as is common with double- 
loop learning. Rather, the important distinction between dialogue and debate is that dialogue calls forth a 
different structure, because of its creative quality. 

Specifically, dialogue does not involve breaking down and examining parts, but allows meaning to flow 
and emerge rather than remaining with a particular point of view. Fundamentally, the dialogue process 
recognizes that one person’s thoughts are not the whole truth (Bohm, 1996) but are instead part of a larger 
truth (Isaacs, 1999). The acts of taking new actions, suspending judgment, listening instead of reacting, 
writing down and reflecting on thoughts, inquiring rather than advocating, and contemplating on the 
underlying value of those actions is when and where the shift in thinking occurs (Argyris, 1994; Shulman, 
1999). This dialogue process based on Bohm’s (1996) theory engenders transformative learning by 
introducing and drawing attention to new ways and reasons for acting differently, reflecting on assumptions, 
and helping to align values to action (Isaacs, 1999). 

A key practice in dialoguing is surfacing assumptions and then suspending these assumptions and 
judgments. Suspending assumptions and judgments does not involve putting them in abeyance, but rather is a 
means to subject them to conscious examination and exploration. In dialogue, as noted by Bohm (1996), the 
intent is to understand others’ points of view and become more open to new ways to perceive and think about 
the situation. We realize our judgments and judgments made by others regarding how we think of things are 
not necessarily the truth. However, they can be seen as true within the author’s situatedness, and the 
challenge is to see how different authors investigate similar problems and reach similar or conflicting 
conclusions, and how this fits together into a coherent pattern. 

In exploring one’s own underlying beliefs, assumptions, inferences, and generalizations, scholars can 
begin to explore the similarities and differences with others and discover commonalities (Simon & Goes, 
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2013). The reviewer proceeds from a place of genuine curiosity and wondering. Focusing on penetrating 
questions with no clear answers opens the way for seeing things more clearly. 

Reflection is necessary for applying triple-loop learning and creating meaning through virtual dialoguing 
with other researchers. Reflection allows us to turn things over in creative ways and improve the quality of 
collective thinking and make a contribution to resolution of a complex problem. 

The following example illustrates application of triple-loop learning to the development of a literature 
re view: 

Caffeine is the most widely consumed psychoactive drug in the United States, but unlike many other such 
drugs it is legal and unregulated. Caffeine is often used by students studying for exams or when they 
need to stay alert in an early morning class. A variety of outcomes have been reported from people who 
consume caffeine (Rogers et al, 2010; Canales, 2010). However there is a lack of clarity regarding the 
efficacy of caffeine intake while conducting scholarly research. A case study of male and female doctoral 
candidates who have completed their course work and are engaged in dissertation writing will be used 
to investigate whether or not caffeine consumption is related to effectiveness in scholarly research. 

In addition to the key words searches used for the single and double-loop review, the search could 
include terms like: stimulants; alertness; wakefulness; motivation; locomotion; social and cultural 
aspects of using stimulants; and means to enhance scholarship. Along with this conventional approach, a 
snowball type of tactic can be employed to find relevant sources. This entails searching the references of 
the articles that were retrieved, determining which of those seem germane to the conversation, finding 
the sources, reading their references, and repeating the process until a point of saturation is reached when 
no new pertinent studies emerge. 

Once the review begins, the goal is to go beyond the narrative, exposition, argument and persuasion (used 
in single and double-loop learning) and focus on solving problems and developing new ideas through critical 
dialogue. Single and double-loop questioning starts with ‘What?”, ‘Who?’, and ‘How?’ Triple-loop 
questioning progresses to such questions as 'Why?' and 'What else is needed?' This involves what Trede, 
Higgs and Rothwell (2008, p.5) suggest as being “open and yet skeptical, being comfortable with ambiguity, 
and extending one's comfort zone by blending deeper with critical perspectives.” This type of review 
represents a collective critical voice rather than the voice of one or two researchers. 

Figure 1 summarizes the process of the triple-loop literature review focused on the caffeine study. 



Figure 1. Triple-Loop Learning and Caffeine Study 
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5. PUTTING THE LITERATURE REVIEW TOGETHER 

In the 21st century, it is unrealistic to expect an exhaustive review or an assurance of locating and 
considering every available piece of research on a certain topic, published or unpublished. The difference 
between the literature review for a journal article and a literature review for a dissertation is primarily about 
breadth and depth. The net for capturing research in a dissertation is generally wider and deeper. The 
expectation in a dissertation is that there will be an extensive review of the literature including the most 
current peer-reviewed studies along with germinal manuscripts in a field. A literature review for a journal 
article tends to consider a representative sample of articles and inferences are made about the entire 
population of articles from that sample (Slavin, 1986). 

The literature review for a dissertation usually begins with an introduction to the study and an explanation 
and justification for the scope of the review. This includes the key word data search used as well as the data 
bases consulted. Each subsection needs to be meaningful, relevant to the problem under investigation, and 
presented in an orderly, logical, transparent, and flowing manner. A detailed description of the procedure 
used to locate relevant studies is needed so that, theoretically, other researchers following the same 
procedures under the same conditions would find the same set of articles. 

The primary audience for a dissertation consists of faculty from the appropriate university, and the 
secondary audience is composed of scholars within the field of the dissertation topic. Primary, peer-reviewed 
sources make up the overwhelming majority of references in a dissertation. Secondary sources, regardless of 
how compelling, are considered hearsay. An historical review, with appropriate citations from germinal 
studies, often follows the introduction. This is where the background of the problem is usually presented. 
Here, and throughout the entire literature review, evidence should be presented that positions the current 
study in comparison with studies found in the search. Discussions should have depth and elucidate a clear 
understanding of why each reference is included. 

Ambiguities need to be addressed and the situatedness of each reviewed study is explained. In a 
quantitative study, a discussion of each variable, and the relevancy of that variable to the study, and how 
variables connect in relationships is usually included. A critical examination of the methodology and 
theoretical framework regarding the problem under investigation needs to be deliberated. 

5.1 Evidence of Triple-Loop Learning 

The demonstration of triple-loop learning is apparent when new perspectives are presented that demonstrate a 
wider domain of literature and synthesis of the literature reviewed with new research. Triple-loop learning is 
indicated when there is a critique of the practical significance and possible resolution of similar problems in a 
variety of contexts, and the need for a fresh perspective is addressed. The writing reflects an open- 
mindedness to ideas, even when the ideas of others conflict or contrast with the researchers. Open- 
mindedness is demonstrated by a) avoiding intensely emotional language, b) exhibiting ontological humility 
in the appreciation for the complexities of a topic c) respecting others in the field, and d) presenting 
arguments that reflect respect and civility. Enacting triple-loop learning requires what Kabat-Zinn (2003) 
refers to as mindfulness or a deep self-awareness, regarding language, assumptions, tacit thoughts, reactions 
and how situatedness affects interactions. Thus, the purposes of triple-loop learning and dialogue in the 
literature review are, as Isaacs (1993) contends: "to create a setting where conscious collective mindfulness 
can be maintained” (p. 31). This type of thinking can generate alternative ideas, practices, and solutions that 
can be held in tension with previous ideas and practices. 


6. CONCLUSION 

A dissertation literature review is a coherent whole. The task is to orchestrate the voices and ideas of many 
into a form of textual unity, where an understanding of the problem as a whole is established by reference to 
the individual parts, and an understanding of each individual part by reference to the whole. Creating an 
innovative literature review creates a solid framework for relating new findings and establishes a profound 
understanding of how new research advances prior research. 
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When triple-loop learning is applied to the review of the literature, there is continual reflection on the 
process as well as the problem under investigation. Attention is paid to the assumptions and values 
(situatedness) revealing how each study could influence the outcomes of other studies. This type of thinking 
allows an exploration of how and why individual and collective cultures both influence and constrain the 
thought processes. In single and double-loop learning, the goal is to produce decisions and come to some 
form of convergence. Triple-loop learning and critical dialogue seeks to reveal the many facets of complex 
issues and is open to the possibility of divergence and a unique understanding of a problem or situation. 
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ABSTRACT 

A case study involving the establishment of a Connectivist massive open online course (c-MOOC) at a college of 
education is presented. c-MOOCs are seen to represent an approach to learning that should be of interest to educators 
preparing their learners (the teachers of tomorrow) for life and work in a knowledge society. The paper differentiates 
between c-MOOCs and courses that are labeled massive, open, and online - MOOCs, but of a different kind and that 
reflect theories differing from Connectivism in most essential respects. Then, it examines the case of establishing a 
c-MOOC at the college using a methodology for analyzing organizational transformation triggered by the adoption of 
computing technologies. Through the narrative analysis of the actions characterizing the implementation of our initiative, 
we have succeeded in understanding how the affordances of MOOCs may subvert the mainstream agenda of an 
organization and its established practices. This understanding is valuable in future plans to establish a MOOC in its 
appropriate context. The rise of the MOOC is relatively young; hence studying how to implement it is also in its infancy. 
The presentation aims to contribute to this research-in-progress by bringing the teacher educators' point of view. 


KEYWORDS 

Connectivism, innovative pedagogy, online learning, open educational resources (OER), PLE/PLN. 


1. INTRODUCTION 

The idea of networked learning as the springboard to achieving the goal of preparing teachers for education 
in the digital age has roots in the theory of Connectivism, according to which learning is seen as occurring in 
the nodes where people, content, and digital interfaces meet. The best known practical application of the 
Connectivist theory of learning is found in the idea of the Massive Open Online Course (MOOC) such as the 
first one that took place in 2008 and was facilitated by Siemens and Downes, who have developed the theory 
of Connectivism, aiming to consider the broad and wide effects of the network society on learning and 
teaching (Siemens, 2005). Connectivism is based on the idea that knowledge is distributed across a network 
of connections, and therefore that learning consists of the ability to construct and traverse those networks 
(Downes, 2007). Connective knowledge is the knowledge that results from connections among properties of 
different entities. As a theory developed in an age of abundant information and connections, Connectivism 
assumes that the learner’s role is not to memorize or even understand everything, but to have the capacity to 
find and apply knowledge when and where it is needed. In line with Bruns’s (2008) concept of ‘produsage’, 
Connectivist learning is also based as much upon production as consumption of content, while the role of the 
teacher is both a novel role - to enable collaborations with and among the learners in order to create and re- 
create content, and a constructivist role - to design interactions in which learners make connections with 
existing and new knowledge resources. Unlike earlier pedagogies, the teacher is not solely responsible for 
defining, generating, or assigning content. 

Connectivism is an approach to teaching, learning, and student assessment requiring radical changes in 
thinking on the part of all stakeholders at the educational institution in which such a course is to take place. It 
is not self-evident that the institution, which has its established content foci, instructional approaches, and 
organizational structure and practices, would immediately welcome courses embodying such departure from 
what has been defined as normative. It is also not at all certain that the proposed participants in Connectivist- 
based MOOCs, whether as instructors or as students, would welcome the change. The MOOC therefore 
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becomes an example of innovation and change, and an object of inquiry into organizational change and 
leadership. The type of learning that has been found to occur in Connectivist MOOCs appears to be based on 
processes that educators wish to encourage in their students in order to prepare them for life and work in the 
21st century. There is no doubt that such a change in conceptualizing learning and teaching should be 
considered in colleges of teacher education; however, there is also no doubt that resistance will present itself, 
as was indeed the case in our own initiative. 

The paper first differentiates Connectivist MOOCs from courses that are labeled massive, open, and online 
but reflect theories differing from Connectivism in most essential respects. Next, an initiative of establishing a 
Connectivist MOOC at an Israeli college of education is described, using a methodology for analyzing 
organizational transformation triggered by the adoption of computing technologies. The paper concludes by 
depicting this analysis as a narrative network constructed from story fragments with potential connections. 


2. TWO TYPES OF MOOCS: AN OVERVIEW 

"MOOCs have been around for a few years as collaborative techie learning events, but this is the year 
everyone wants in", says a recent New York Times article*. "MOOCs (Massive Open Online Courses) are the 
educational buzzword of 2012", adds Sir John Daniel (2012). The media hype about MOOCs in higher 
education has focused on their massive scale; however, the real revolution - as Daniel puts it - is that 
"universities with scarcity at the heart of their business models are embracing openness" (Daniel, 2012). 
From a pedagogical point of view, the MOOC phenomena redefines what is meant by “learning,” “teaching,” 
and “assessment,” and at the same time blurs the boundaries between them. 

The first MOOC took place in 2008 as an open online course at the University of Manitoba, Canada. The 
course, Connectivism and Connective Knowledge (CCK08) was facilitated by George Siemens and Stephan 
Downes, who have been developing the pedagogical theory of Connectivism and have regarded MOOCs as 
practical implementations of their theory (Siemens, 2012). The term itself was coined by Dave Cormier who 
joined in facilitating several other MOOCs, including PLENK2010^ that has been described as “a 
conglomerate consisting of various layers: live sessions... recordings... a complexity of discussion forum... 
the course Wiki and Blog... and the unique course aggregator named the Daily” (Levy, 2011). MOOCs of 
that type were later labeled “Connectivist MOOCs” (c-MOOCs), to distinguish them from the current wave 
of MOOC offerings that share a little with Connectivist pedagogy. It is the purpose of this section to make 
this distinction clearer by elaborating on Downes (2012) terminology of c-MOOCs versus x-MOOCs. 

2.1 Connectivist MOOCs (c-MOOCs) 

Learning in Connectivist-based MOOCs reflects processes necessary for life and work in a global networked 
world, in which information is characterized by rapid change and renewal, is collectivized, poorly organized, 
and incompletely evaluated (Kop & Hill, 2008). The challenge is for each learner to construct a personal 
learning network (PLN), by eliciting what is personally meaningful from the network of information and 
interactions. Such learning is “...highly social. The learning comes from content presented by a lecturer, and 
then dialog via social media, where the contributions of the participants are shared” (Quinn, 2012). 

In addition to the abovementioned CCK08 and PLLNK2010, two noteworthy c-MOOCs are the eight- 
month long Change 11^ and the short six- weeks MOOCMOOC"*. 

These c-MOOCs are revolutionary in that they erase existing boundaries between the institution and the 
world “outside” it. Such Connectivist-based MOOCs call into question academic responsibility and 
institutional accountability. However, the seeds of the MOOC that were first spread as practical 
implementations of Connectivist theory have been supplanted by others, which have developed into a 
different "flower" entirely, as the next section details. 


’ "The Year of the MOOC" http://www.nytimes.eom/2012/l 1/04/education/edlife/massive-open-online-courses-are-multiplying-at-a- 
rapid-pace.html?pagewanted= 1 &_r= 1 

^ PLENK2010 - Personal Learning Environments, Networks and Knowledge, http://connect.downes.ca/ 

^ Change: Education, Learning, and Technology! (Eall 2011). 

MOOC MOOC a MOOC about MOOCs (Summer 2012). More Connectivist MOOCs offered since 2008 can be found in 
http://mooc.ca/. 
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2.2 Other Types of MOOCs (x-MOOCs) 

Right until the fall of 2011, the term “MOOC” was not used much by educational technology scholars and 
was not acknowledged at all in the mainstream public discourse. Those who did mention the term 
unequivocally denoted a practical application of Connectivism, as has been briefly discussed above. 

The turning point seems to be with the Artificial Intelligence experimental open online course offered in 
the fall semester of 201 1 by two well-known computer scientists from Stanford. The first wide publication of 
this course in the New York Times (August 15, 201 1) ^ didn’t even mention the term “MOOC”. Six months 
later, Quinn first distinguished between two types of MOOCs: the Stanford model and the Connectivist 
model. The goal of both types, writes Quinn, is to enable a free and “high quality learning experience to 
anyone with sufficient technical ability and access to the Internet”, but as opposed to the social nature of the 
Connectivist model, in the Stanford model “the experience is, effectively, solo”®. 

During the spring of 2012 the wave turned into a Tsunami. Numerous news articles, blog posts, media 
interviews, and social networks posts flooded the Internet with new MOOC announcements, calls for 
participation, and critiques. Within a few weeks, MIT announced MITx; a consortium of Ivy League 
universities including Stanford and Penn State established Coursera; and Harvard University joined forces 
with MIT to create EDx - to name only a few. In March, Hill^ wrote that “there are really two variations of 
MOOCs with quite different approaches - witness the Stanford and MITx version vs. the rhizomatic 
version”. While the “O” that stands for “open” is thought to be the dominant letter in the original 
Connectivist branch of MOOCs, “M” seems to be the dominant letter in the Stanford branch. The most press 
cover, however, has been based on the Stanford model of MOOCs. In a blog post in July 2012, Downes 
(2012) therefore proposed a new terminology: x-MOOCs like Udacity, EdX, Coursera; and c-MOOCs - 
Connectivist MOOCs - providing not only open online content in a domain but also immersion into a 
community of practitioners associated with that domain. Today, while x-MOOCs clearly dominate, c- 
MOOCs are also spreading around the globe and the variety of subjects they deal with. Both types provide 
new models for learning at a time when traditional school learning is widening the rift between learners’ 
experiences in and of the world and their experiences in formal school settings. 

2.3 c-MOOC as a Pedagogical Innovation 

The type of learning that has been found to occur in MOOCs of both types appears to be based on processes 
that educators wish to encourage in their students. Anderson (2008) pinpoints two major forces shaping the 
knowledge society: “greater intercultural interaction, enabled by global electronic networks, and an economic 
system in which knowledge functions as a commodity” (p. 7). In the face of such a “given,” the global citizen 
needs to learn how to construct knowledge and develop adaptability, the ability to work in teams, and skills 
relating to the retrieval, organization, and critical evaluation of information (Mioduser, Nachmias, & 
Eorkush-Baruch, 2008). 

Our work, started prior to the current public interest in the MOOC phenomena, is based on the 
Connectivist vision of the c-MOOC. We see this model as aiming to bring about change and innovation on a 
number of levels: 

Pedagogical - with a redefinition of what is meant by “learning”, “teaching”, and “assessment”. The 
redefinition of pedagogy will affect learners and teachers alike. 

Content - once a traditional course (even a traditional online course) becomes a c-MOOC, it demands 
deep-level revision of content. In addition, as the content is distributed and takes on a “life” of its own, 
independent of its point of origin, a c-MOOC necessarily involves the erosion of traditional boundaries 
regarding content creation and development. 

Technological - c-MOOCs are founded on technologies that encourage interaction between people, 
people and content, and people and interfaces. 

Organizational and cultural - the c-MOOC instructors have to collaborate in ways that they have 
probably not before experienced and restructure their courses. 


^ Markof, J: "Virtual and Artificial, but 58,000 Want Course". New York Times - Science. 
http://www.nytimes.com/201 l/08/16/science/16stanford.html?_r=2&ref=technology 
^ Quinn, C: MOOC reflection. Posted February 29, 2012 in http://blog.learnlets.com/?p=2562 . 

^ Hill, P: E-literate blog, http://mfeldstein.com/moocs-two-different-approaches-to-scale-access-and-experimentation/ 
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It is against this background that a c-MOOC was seen by the authors as representing suitable preparation 
for developing, not only specific content knowledge, but also the 21st century literacies noted above. The 
initiative arose from our shared conviction that it would provide a model for learning at a time when 
traditional school learning is widening the rift between learners’ experiences in and of the world and their 
experiences in formal school settings. We therefore initiated a conversation in our college with the aim of 
finding a suitable framework within which to establish a c-MOOC for our students - the teachers of 
tomorrow. The next section describes the first steps in our ongoing effort. 


3. THE JOURNEY TO IMPLEMENTATION 

We present the experiences characterizing the journey to establishing a c-MOOC at our institution as a 
narrative network (Pentland & Feldman, 2007), based on personal reflection and analysis of interviews with 
stakeholders whom we identified as potential partners in our initiative. 

The analysis that follows is based on Pentland and Feldman’s (2007) characterization of a narrative 
network “as a device for representing patterns of ‘technology in use’.” (p. 781). Pentland and Feldman use 
the term network “to draw attention to both potential and realized interconnections between actants and 
actions and the fluidity of these interconnections” (p. 781). The “narrative” aspect is rooted in a philosophical 
perspective that “different interconnections make different stories” (p. 781). This approach has roots in actor- 
network-theory (ANT) (e.g. Latour, 2005). 

3.1 Enactment of Organizational Eorms in a Narrative Network 

This section presents a number of stories centered on meetings we had with various potential partners in our 
organization. Each story involves a number of actants: the authors, the potential partners, and the idea of the 
MOOC with its affordances. In ANT, actants include both human players (actors) and non-human entities 
such as an idea, a tool, a computer interface, etc. (Latour, 2005). An affordance refers to the possibilities 
latent in any part of the environment vis a vis an agent. Gibson (1977), who first coined the term in relation 
to animals interacting with their environment, defines an affordance as the opportunities for action provided 
by a particular object or environment. Norman (1988) applied the concept of affordances to understanding 
people’s interactions with everyday things and computer interfaces alike. Just as an everyday object like a 
door handle offers possibilities for opening the door by turning it while simultaneously pushing or pulling the 
door (and something in the design of the object will hint at its use), so computer interfaces should be 
designed in such a way that their use is suggested to the user. 

What is specifically relevant to the present paper is that the concept of affordances foregrounds the notion 
that things - in our case, the c-MOOC - can be characterized by a “psychology” or that they have embedded 
within themselves properties for action. However, as Kirkeby (2003) points out: “Affordances are emergent 
properties of objects in the environment but only in relation to actors through potential activities” (p. 10). 
This makes affordance theory - especially later emphases on the relational aspect of affordances- compatible 
with other theories that consider the potentials of “things” in relation to subjects (human actants in ANT) and 
not only as having properties for action in themselves. 

The narratives that emerged from our interactions with people whom we identified as potential coalition 
partners in the establishment and implementation of the c-MOOC reflect how such an initiative involves an 
organizing of people in relation to a technology. In our case, the potential affordances of the technology at 
the center of the initiative were seen as having a possible destabilizing influence on the existing practices of 
the organization. 

The narrative methodology - itself mirroring many aspects of Connectivism (in the broad sense of 
emphasizing connections and networks) - enables us to compile stories told from different perspectives, 
based on our encounters with the potential partners we contacted, and to trace actions and reactions when the 
idea of the MOOC and its affordances were placed as the focus of the discussion. Each story presents the 
perspective of at least one potential partner in interaction with us (the authors) as initiators of the MOOC 
idea. This methodology is reflected in Pentland and Eeldman’s (2007) observation that “anything that 
influences the ‘plot structure’ is organizationally significant” (p. 784). 
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3.1.1 Story 1: Don’t Shake the Ground 

Shortly after developing our first plan of action, it became clear from a discussion with Anna - one of the 
department heads* - that the specific context was not acceptable since the existing course had itself 
undergone numerous transformations and had only that year achieved the goals that had been set for it. The 
instructors and students had reported being satisfied with the newly transformed course so any additional 
transformation - especially one requiring a total change of direction - was not perceived as appropriate at the 
time. The arguments against setting up the c-MOOC in the proposed context were expressed as follows: “The 
instructors have been through enough changes in the development of the existing course, and they will be 
unwilling to change things again, especially if we are talking about such a radical change of emphasis.” 
(Anna Interview, September, 2011). 

These arguments reflect the tension between the potential transformations the c-MOOC initiative might 
bring about, and the existing organizational practices. However, alongside the arguments against establishing 
the proposed c-MOOC, Anna was in favor of the general idea of establishing a c-MOOC at the college and 
encouraged us to pursue the initiative in relation to an alternative disciplinary field. We therefore 
reconsidered the plan and decided to direct our efforts to finding a more appropriate context. 

3.1.2 Story 2: Openness in Three Acts 

This story is mainly based on a meeting with Beth, coordinator of the Social Involvement program of the 
college (May, 2012). The narrative constructed here cuts across at least three sub-plots, each a few months 
apart from the others. Most of the actants (people and technologies) were the same, or they referred to one 
another across the story plots. The connection between the chronologically first narrative and the two later 
ones became apparent to the authors only a short while before the meeting on May 2012. For the sake of 
clarity, the narratives are presented chronologically as Stories 2-A, 2-B, and 2-C. The reflections on these 
connections are deliberately presented as interruptions to the chronological sequence since it is the reflections 
themselves that helped us put together the analysis. 

Story 2-A: Vision of an Open Blog 

The first sub-plot took place about a year before the MOOC initiative was even conceptualized by the 
authors. It involved a proposal by one of the lecturers in the program, Jake, to set up an open blogging 
environment for one of the courses in the Social Involvement program. In retrospect, the blogging 
environment was to include many elements that also characterize MOOCs, but these were not identified as 
such by any of the people involved. This narrative is presented in detail below - as a flashback in the 
framework of the third (Story 2-C). 

Story 2-B: Vision of an Open Learning Environment 

The second sub-plot is set in March 2012 when the authors approached Beth with a proposal to set up a 
c-MOOC in order to bring together various inter-linked aspects of the courses comprising the program. In our 
search to pinpoint an appropriate context, we decided that the openness of such an online environment might 
answer many of the needs arising from the courses. 

Each of these courses contains both a theoretical and a practical component, and as communicated to us 
by Beth, what was needed was some way of connecting between the theoretical and practical components of 
each course, and between all the courses pertaining to the program at the college. 

This potential connection between all the courses formed the focus of the meeting between us - the 
authors and Beth - in March 2012. We presented the theoretical background underlying MOOCs and 
explained how such an online environment could help to create connections between the various courses, and 
between the theoretical and practical aspects of the program. Beth clearly expressed interest but requested 
that we continue to flesh out the idea and return to her with a more developed proposal that she could 
introduce to the course lecturers at a later date. 

A few weeks passed, during which we reassessed the objectives of the c-MOOC initiative and came up 
with a plan. During this period, we met with the lecturer whom we considered would be a potential partner in 
the revised initiative, Jake, and realized that our initiative shares many similarities to the one he had 
presented a year earlier (Story 2 A). 


* Pseudonyms are used to refer to the various stakeholders or potential coalition partners. 
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Story 2-C: Openness vs. Control 

The setting for the third sub-plot was a second meeting with Beth on May 2012. Beth was asked to recall the 
prior initiative (Story 2-A). This initiative - sharing similar characteristics to our conception of a c-MOOC 
although it had not been described as such at the time - had essentially been rejected. It is interesting to 
consider, in retrospect, the basis for the rejection since what happened then sheds light on the factors to be 
considered when proposing such an initiative within the educational establishment. 

In her reconstruction of the earlier meeting, Beth mentioned that it had involved a number of people 
whose positions could have made them potential partners in the endeavor, including Jake. She mentioned that 
one of the main objections was that the open blog envisaged by Jake would bypass the college’s official 
Internet site, and that the issue of locus of control was also voiced by various attendees. Although, in 
retrospect, the authors consider the issue of locus of control to be the main one underlying resistance to the 
establishment and implementation of a c-MOOC in an institute of education, the objections at the earlier 
meeting focused mainly on the issue of the website as a marketing conduit. 

3.1.3 Story 3: c-MOOC as a Disruptive Innovation 

The actants in this story are the authors themselves. Awareness of the broader contextual implications of 
establishing a c-MOOC occurred when the authors began to consider how a c-MOOC, almost by definition, 
will break the boundaries of the institution that gives birth to it since the locus of control moves from the 
institution, or from the lecturer who is an official representative of the institution, to the students and people 
outside the institution itself: “It is clear now that MOOCs can turn education on its head. Control is no 
longer with the teacher or teaching agent as in behaviorism, or with the learner, as in constructivism, but 
distributed, everywhere, and nowhere...” (Authors meeting. May 2012). 

Awareness of the potential tension between the affordances of the c-MOOC and the institution’s 
organizational culture took place at a particular moment in the meeting, when discussing the objections that 
had been raised in Story 1 and when considering how to revitalize the initiative described in Story 2. It was 
then that we realized that c-MOOCs may subvert the organization’s agenda by placing the locus of control 
outside the boundaries of the organization: “A MOOC goes beyond the time and space barrier reflected in 
traditional pedagogy... It breaks the barriers between the natural technological living space of the learner and 
the LMS set up by the instructor.” (Authors meeting. May 2012). 

In an educational organization, “established” courses are those around which there is consensus by the 
“establishment.” The educational establishment may be understood in the broad sense of what is accepted by 
the society’s education system, and in the specific sense of what is accepted by the specific institute of 
education. Since the implications of these conclusions were far-reaching, we decided to verify them and 
approach Beth again, as well as additional persons whom we identified as actants in these events. This led to 
the narrative presented above in Story 2-C. 

3.2 Connecting the Threads of the Narrative Network 

The stories presented above would remain narrative fragments (Pentland & Feldman, 2007) unless a 
deliberate attempt is made to show how they constitute part of an organizational network. They point out: 
“Actants are connected through actions into narrative fragments” (p. 789). Each of the stories presented in the 
previous section is, indeed, a narrative fragment. However, they also observe: “Narrative fragments are 
connected with one another in the construction of narratives” (p. 789). They illustrate - and visually depict - 
how it is possible to construct a narrative network out of a number of narrative fragments in the context of 
their own case. Fig. 1 was constructed in accordance with their example and depicts the narrative network of 
our MOOC initiative. It is possible to see how parallel stories involving the same actants gained coherence as 
the authors connected the fragments into a single network. 
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Eigure 1. The Narrative Network of the MOOC Initiative 

The narrative network can help to identify fragments that might get associated with other fragments in 
efforts to change organizational practices. For example, narrative fragment 7, referring to the authors’ 
awareness of what a MOOC really means in an educational organization, can be foregrounded in any 
subsequent discussions on moving forward the MOOC initiative. It can be juxtaposed with narrative 
fragment D, to exemplify how a MOOC initiative can be diverted in directions that, albeit compatible with 
the organization’s existing practices, contradict the organization’s vision and purported practices. The 
contrast between these two narrative fragments parallels the distinction made by Pentland and Feldman 
(2007) between the ostensive and the performative aspects of any organizational routine. Whereas the 
performative aspect refers to actual practices, the ostensive aspect refers to the participants’ awareness and 
understandings of these practices. 

Through the narrative analysis of the actions characterizing the implementation of our MOOC initiative, 
we have succeeded in understanding how the affordances of MOOCs may subvert the mainstream agenda of 
an organization and its established practices. This understanding, reflected in a node connecting between a 
number of narrative fragments comprising the stories presented earlier, is valuable in future plans to establish 
a MOOC in its appropriate context. 


4. SUMMARY 

In this paper we have followed up on the roadmap for establishing a c-MOOC at our College of Education. 
Our experiences surrounding the initiative was comprised of meetings with stakeholders whom we identified 
as potential coalition partners, interviewing some of them, and reflections between us that arose from 
discussions with them and with others. Applying narrative network methodology to make sense of the events, 
the experiences are presented as a number of stories whose inter-connections become apparent following 
narrative analysis. The analysis has raised significant questions regarding the organizational context in which 
a MOOC may be implemented and has implications for understanding organizational transformations in light 
of technological innovation. 
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ABSTRACT 

It is common to see students multitasking or switching between different tasks on the computer while also listening to the 
teacher lecture in the front of a classroom. In today’s classrooms, students have much greater control over how they use 
their time, with the classroom integration of computers and mobile devices combined with social media and text-based 
chat tools. It is important to understand how students learn with the connected technologies. This study examines high- 
school students’ ability to take notes and obtain information from a video while simultaneously carrying a conversation 
online with a friend. 


KEYWORDS 

Cognition; learning; digital technology; multitasking; chat 


1. INTRODUCTION 

This study examines issues related to media multitasking and learning in educational environments. The rise 
of connected devices in the classroom has increased the level of multitasking during lectures and lessons. 
Students are often observed taking notes while texting a friend on their phone or updating their Facebook 
page. Students can be observed performing these types of dual-tasks throughout the day and it has become a 
natural part of how they function. What is the cognitive cost of this practice? 

Poldrack and Foerde (2007) found that people had a harder time learning new things when their brains 
were distracted by another activity. The Functional Magnetic Resonance Images (fMRIs) used by researchers 
showed that when people learned without distraction, an area of the brain known as the hippocampus was 
involved. This region of the brain is critical to the processing and storing of information. However, the 
hippocampus was not engaged when people learned while multitasking. Instead, the region of the brain called 
the striatum was activated. The striatum is activated by stimuli associated with reward or by stimuli 
associated with aversive, novel, unexpected or intense experiences (Schultz, 2010). Results indicated that 
learning while distracted or multitasking would alter the brain’s learning processes and change the way 
people learn (Poldrack & Foerde, 2007). Foerde, Knowlton, and Poldrack (2006) found that learning new 
things is dependent on working memory where as habit learning is not as sensitive to working memory. Some 
tasks such as learning new skills may require high cognitive loads, while other familiar and automatic tasks 
may require lower cognitive loads. Lin and Bigenho (2012) examined undergraduate student’s memory recall 
under nine conditions in a 3x3 study with three levels of environmental distraction and three levels of 
note-taking. The study found significant interactions between environment and method of note-taking as the 
computer seemed to mediate the effect of an auditory distraction during computer aided note taking. The 
study also found that students were more successful when taking notes at lower levels of distraction and that 
those who did not take notes consistently performed poorly on the recall tasks regardless of distraction level. 
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2. THE STUDY 

This study was designed to examine high-school students’ abilities to take notes and obtain information of a 
video while carrying a conversation online with a friend at the same time. The experiments were designed to 
imitate the practice of chatting online with friends while attending a classroom lecture. Two scenarios were 
created: one was to have students watch a video and take notes of the video only; another was to have 
students watch a video and take notes of the video while chatting with a friend. Specifically, students 
observed a 20-minute video lesson and were told they would be tested on the content of the video after each 
lesson in both scenarios. The students were also encouraged to take notes following a format and method 
most comfortable to them. The difference lies in the online chat; in one scenario, the students focus on the 
video and take notes only; in another scenario, the students maintain a chat with an Artificially Intelligent 
(AI) agent while observing the video and taking notes. Two videos of the same length and difficulty level 
were used. All students participated in both scenarios. In addition, four conditions were created so that the 
sequence of the videos and the sequence of the chat/no chat situations would not affect the results of the 
study. Data were collected from the student participants’ quiz scores of the videos, their notes, and their chat 
transcripts. The participants were also asked to fill out an open-ended questionnaire to describe their 
experiences with the experiment and their learning habits. The study employed both quantitative and 
qualitative measures following a repeated measures design. 

Questions we hoped to answer through this approach include: 

• What were the patterns, if any, to the distribution of notes as compared to the presentation of the 
video lecture and the level of chat activity during the chat condition? 

• What were the patterns, if any, in the answers on the quizzes as related to the flow of the 
lectures, the notes that were taken, and the level of chat activity in the chat condition? 

• How might the lecturer’s style and method of delivery affect the notes recorded, the level of chat 
activity or the participant’s ability to recall information? 

• What does this indicate about the nature of the topics and the methods of delivery of the 
information in increasingly complex and connected classrooms and meeting centers? 

2.1 Participants 

Participants in this study were from a small to mid-sized independent high school in the Southwest United 
States. An a priori analysis was conducted from the initial study and it was determined that 40 to 54 
participants would allow us to detect a medium effect of f = 0.25 with the alpha level set at p < .05. A total of 
39 students participated in the study. They ranged in age from 15 to 18 with the average being 17 years. All 
participants were in 9*, 10*, 11*, or 12* grade. Participant consent and assent forms were obtained for each 
participant. 

The experimental conditions, data sources and design are summarized in table 1. This format allows for 
the following comparisons: video presentation order (First::Second); video topic (Video Topic 1:: Video 
Topic 2), and Chat (Chat::No Chat). Each participant was assigned to a condition and completed all tasks 
related to that condition. There are two scenarios (Chat/No Chat) which are presented in a balanced format to 
help identify any effects of video content and presentation order. 


Table 1. Sequence and Number of Participants in Each Condition 


Conditions First video and quiz 

Second video and quiz 

Survey 

Number of participants 

Condition i Video A (No Chat) 

Video B (Chat) 

Post Survey 

10 

Condition I Video A (Chat) 

Video B (No Chat) 

Post Survey 

10 

Condition ( Video B (No Chat) 

Video A (Chat) 

Post Survey 

10 

Condition I Video B (Chat) 

Video A (No Chat) 

Post Survey 

10 


2.2 Data Analysis 

The study generated several different artifacts as data and consisted of both qualitative and quantitative data 
forms. Additionally, the video lessons provided a time code that could be aligned with each data type 
allowing for all data to be examined across the temporal continuum of the video lesson. 
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Video Transcript: Each video transcript was organized into sentences with the video on happiness 
resulting in 256 sentences compared to 184 for the video on energy. This allowed the note and quiz data to be 
mapped directly back to the video transcript using what we called knowledge units. Knowledge units are a 
count of specific mappings from the notes to specific sentences. 

Notes of the videos: Participants took notes during each of the video lectures recording content from the 
video lectures. Notes were taken on a computer or on paper as decided by each participant. Notes were coded 
according to the knowledge units they contained. Each knowledge unit was mapped to specific sentences 
from the transcript providing a temporal scale to the notes in relation to the content of the video. Data were 
then loaded into Atlas.TI for future analysis of patterns. A scaled score for comparison between videos was 
created by dividing the total number of knowledge units recorded for each set of notes by the total number of 
knowledge units (sentences) in each transcript. This value was multiplied by 100 resulting in a scaled score. 

Post video lecture quiz: At the end of each video lecture, the participants were given two minutes to 
review their notes and then were asked to take a short quiz on the material from the lecture. The questions 
were directly related to material presented in the lecture they had just observed. The quiz results were 
subjected to an item analysis and then the results were mapped back to the video’s temporal scale and 
compared with the analysis of the temporal analysis of the notes taken. 

Chat transcript: Chatting with the AI chatbot created a transcript of the exchange. Each chat transcript 
was archived preserving the content of the conversation. Each transcript has a time code embedded and all 
chats were all initiated with the same method so that they were aligned with the time-code of the video 
lecture. The units of analysis for the chat were the individual transmissions and frequency of transmission 
mapped to the video temporal scale. Chats were also analyzed for content and any emerging trends. 

Post experimental survey: This was an open-ended questionnaire that all participants completed at the end 
of each experiment. The survey asked the participants about their experience during the two experimental 
trials. There was no timestamp for this data. The post survey results were subjected to content analysis using 
the word as the unit of analysis. 


2.3 Result 

Using a paired t test, the analysis of the chat/nochat conditions indicated significant differences for quiz 
scores in the two conditions t(37) = -5.12, p<0.001 as well as significant differences for knowledge units 
recorded in notes t(37) = -4.02, p<0.001. Additionally, the analysis of knowledge units recorded for each 
video topic also indicated significant differences t(37) = 3.53, p=.001. There are no significant differences for 
the presentation order of videos or chat. Additionally, quiz results based on video topic indicated no 
significant differences. 

Table 2. Table of means for comparisons. Items reaching levels of significance using paired t-tests are indicated with **. 


Data Type 

Comparison Condition Mean S.D. 

Quiz Scores 

. Happy Video 11.28 2.33 

Video Topic Energy Video 10.74 2.28 

Eirst Video 11.39 2.07 

Video Order Second Video 10.64 2.49 

Chat Condition** 

No Chat 12.15 1.59 

Mapped 

Knowledge Units 

Video Topic** Happy Video 9.75 5.53 

viaeo topic Energy Video 13.75 6.40 

^ , Eirst Video 13.12 6.06 

Video Order Second Video 10.41 6.28 

Chat Condition** 

No Chat 13.95 6.39 


Comments from the post experimental survey indicated elevated intrinsic interest in the happiness video 
compared to the energy video while the number of knowledge units recorded in participant notes was higher 
for the energy video . This may have been linked to less familiarity or comfort with the energy topic and 
therefore the perceived need to record more notes. However, since the difference in quiz scores between the 
two video topics did not reach significance, we can treat them as equal when comparing them in the two chat 
conditions. Additionally, we can ignore video order as it did not reach significance. 
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Therefore, when students were in the chat condition, they performed significantly worse on the quiz 
immediately following the lesson, indicating that chatting during the lecture had a significant effect on their 
ability to code the information and commit the information to short-term memory. Since students were also 
allowed two minutes to review their notes prior to testing, it would seem logical to believe that their notes 
were also significantly different between the chat and no chat condition. This conclusion gets increased 
support when examining knowledge units mapped from notes taken under each chat condition. Students 
recorded more knowledge units when not in the chat condition than they did when chatting. 

The large standard deviations in the findings related to mapped knowledge units were likely the result of 
some students taking minimal notes during the lecture and not taking notes at all during the chat condition 
even though they were instructed to do so. Some initial qualitative analyses of the notes and chat transcripts 
indicated that: 1) there were differences in the use of language between notes taken when chatting and 
without chatting, 2) there were cases of chat transaction ending up in notes, 3) the chat transactions may 
serve as indicators for the level of engagement. 

The initial results showed that performing a secondary task such as chatting during a video lecture: 1) 
may reduce the volume of notes taken by over 30%, 2) negatively affected the participant’s short-term 
memory of the video content, and 3) may change the notes by adding multiple sources including the chats 
into the notes on the video content. The complete data results will be reported at the conference. 


3. CONCLUSION 

The number of portable connective devices continues to increase in classrooms resulting in a need to 
understand the effects of using these devices while attending to lessons in the classroom. Also, as the number 
of online learning opportunities increase, so does the access to opportunities to perform multiple tasks during 
a lesson. This includes both educationally designed uses of these technologies for learning as well as the non- 
instructional student mediated use of these technologies. Quantitative results indicate that student’s 
short-term recall was negatively impacted with student-mediated use of these technologies along with the 
amount of information that was actually captured in student notes while chatting during a lecture. We are also 
performing a temporal analysis where we will be able to examine trends that emerge from across the different 
data strands. This type of temporal analysis could provide a further window into what is going on behind the 
numbers and help us understand why some information is retained while other information is not as students 
attend to media multitasking during lectures. We hope that this specific contribution to the field will help 
provide evidence that can inform the effective use of these technologies as well as lesson design and delivery. 
At the end of each trial, participants completed a short assessment to see what they retained in short-term 
memory. While long-term memory is the objective of any learning endeavor, we believe that short-term 
memory is on the path to long-term memory and must be understood in the context of multitasking during 
learning activities. 
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ABSTRACT 

This proposal reports the findings from interviews with construction science students about their visualization of 
problems in a two-dimensional and a three-dimensional task. The specific research questions focused on the individual 
characteristics that students identify as influencing their spatial ability to generate internal and external representations. 
Thematic analysis was used to analyze interview data. The findings showed that the existing instructional model about 
generating representations internally and externally may need revisions to its sequence and more focus on integrating 
students’ prior knowledge and experience. 
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1. INTRODUCTION 

Spatial thinking is an important skill set for various careers, such as medicine, chemistry, engineering, 
mathematics, geography, and architecture (National Research Council [NRC], 2006; Orion, Ben-Chaim, and 
Kali, 1997). Engineers and architects think spatially when designing a new building or roadway. Scientists 
think spatially when looking at complex molecular structures. Therefore, spatial thinking should be included 
in instruction to develop students’ domain expertise. The National Research Council (2006) defines spatial 
thinking as a combination of 1) spatial abilities, 2) spatial skills, and 3) spatial reasoning holistically applied 
to complete a task or solve a problem. According to the Merriam-Webster Dictionary (2012), ability is 
defined as a natural aptitude or acquired proficiency and skills are defined as a developed ability. Spatial 
reasoning involves the application of spatial ability and skills to go beyond what an individual perceives in 
space to reason about an object. To accomplish spatial reasoning an individual must manipulate an object by 
adding to, and transforming, the information input from one’s perception about the object to make decisions 
and solve problems (Tversky, 2005). 

Important to spatial thinking is the ability to generate internal and external representations to assist with 
remembering, understanding, reasoning, and communication about the properties of and relations between 
objects represented in space (NRC, 2006). Internal representations are cognitive whereas external 
representations are graphic, linguistic, or physical. External representation refers to sketches and physical 
models, while internal representation refers to mental images and models. Previous research indicates that 
external representation leads to significantly more successful problem handling, efficient problem-solving, 
and higher quality solutions (Romer, Leinert, & Sachse, 2000). The ability to generate internal 
representations is also important for spatial thinking (NRC, 2006). Individuals generate internal 
representations through a combination of visual images and mental models based on the description, graphic, 
or picture provided about the subject (Schnotz, 2005). 
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2. PURPOSES 

Past research on spatial reasoning did not address ill-structured problem solving in the context of a science, 
technology, engineering, and mathematics (STEM) discipline (Carroll, Thomas, Malhorta, 1980). 
Ill-structured problems are those with vague goals and multiple solutions, or solution paths, and solving this 
kind of problem requires problem representation, justification skills, monitoring, and evaluation (Ge & Land, 
2003). The goal of this research was to investigate students’ spatial representation ability in different 
dimensions. Specifically, the purpose of the study was to 1) investigate the approaches taken by students 
when solving a spatial problem-solving task in two-dimensional (2D) context versus a three-dimensional 
(3D) context; and 2) to understand how internal and external representations interact with students' 
approaches to the spatial tasks. The current study sought to answer the following research questions: 

1 . Which individual characteristics and views about visualization do undergraduate students identify as 
influencing their ability to generate internal and external representations? 

2. How do individual experiences and prior knowledge influence an undergraduate student’s ability to 
visually represent problems internally and externally for spatial problem solving? 

3. How do undergraduate students approach a spatial-related task when it is presented in a two- 
dimensional view compared to the same task presented in a three-dimensional view? 

4. How do novice undergraduate students visualize a spatial task presented in a two-dimensional view 
compared to a spatial task presented in a three-dimensional view? 

o How do students represent each spatial-related task internally? 

o How do students represent each spatial-related task externally? 

The findings from this research would serve to inform the instruction involving spatial reasoning to 
determine if the current instructional strategies about spatial representations are appropriate or sufficient. 


3. METHOD 

3.1 Participants and Sampling 

An exploratory study was conducted with 43 undergraduate students (37 male; 6 female) enrolled in a 
Commercial Construction Capstone course in a College of Architecture at a mid-western United States 
university in two consecutive academic years. The written tasks in this study were completed as course 
activities by all the students. However, only thirty (29 male; 1 female) volunteered to participate in a 
follow-up interview. At the time of this paper 11 semi-structured interviews had been completed with 10 
male students and 1 female student. Recruitment of students, signed consent to participate in the study, and 
administration of tasks was performed by the co-principal investigator for the study. The course instructor, 
who also administered the task, was the study’s principal investigator and conducted all the interviews. 
Pseudonyms were assigned to each interview participant for privacy purposes. 

3.2 Data Collection 

This research was designed as a qualitative study. Two spatial-related problem-solving tasks (2D and 3D) 
were collected. In Task 1 participants were asked to identify the errors present in the building system using 
the 2D views provided. Errors were defined as anything that would impede or prohibit the physical 
construction of the building element. The participants were asked to first identify all the errors in the 
drawings by placing a number on the element in which the error occurs, then transfer each number to the 
Errors Table and include an explanation of the error to justify their identification of the element as an error. 
The participants were also given the option to generate a 3D representation from the 2D views on the blank 
sheet of paper provided with the study materials. Task 1 took twenty minutes. Seven calendar days after Task 
1, Task 2 was administered, with the duration and instructions identical to Task 1. The only difference 
between the two tasks was the drawings provided for Task 2 were 3D views of the building instead of 2D 
views. 
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Interview durations ranged from 25 to 45 minutes. Each semi-structured interview consisted of 17 
questions, such as “What means of visualization did you use while working on the 2D error identification 
task? ’’ and “Describe any additional ability, knowledge, techniques, or experience that you think contributes 
to your use of visualization. ’’ 

3.3 Data Analysis 

The researchers followed the guidelines and procedures recommended by Shank (2002) to making sense of 
the interview data. The procedures for searching were 1) basic themes or patterns in the data, 2) plausible 
explanations through comparison and contrast, and 3) variables or factors for connections. Interview 
transcripts were analyzed and the data reduced through four steps until five themes emerged from the data: 

1. Visual Selves -Individual beliefs and perceptions about visualizing. 

2. Approach to Task - Students describe of their approach to completing the task. 

3. Influences on Visualizing Capabilities - Students describe influencing factors on their individual 
capabilities to visualize. 

4. Limitations/Difficulties to Visualizing - Students describe perceived limitations and difficulties to 
visualizing. 

5. Benefits of Visualizing - Students describe their perceived benefits to visualizing. 


4. FINDINGS 

Due to the space limitation, only three representative interviews are reported in this paper; in addition, this 
paper only focuses on the theme “Visual Selves -Individual beliefs and attitudes about visualizing”. 

Based on the participants’ responses the most positive influences on students’ ability to generate 
representations are ability, prior knowledge, and related work experience. However, participants identified 
individual characteristics as having the most influence on their creation of internal representations. Each of 
the participants discussed visualization in some detail and it emerged as a recurring theme throughout the 
data. It was commonly agreed that one must be able to create an internal representation before creating an 
external representation. 

Gerald and Donnie each expressed their belief that you have to have an internal representation, or mental 
image, of what you are going to sketch before you start drawing. Donnie who said: 

“I think once I got going with an external representation I would be better. Once I started 
visualizing in my head it would make me understand everything else a little bit better before I 
started creating that external representation. By the time I got through understanding the image in 
my head then creating a sketch would be good.” 

Interestingly, the students apparently believe that the ability to create internal representations is a natural 
ability rather than something that can be taught. Eor example, Gerald, a senior in the Construction Science 
undergraduate program, and a student who has had prior instruction about visualization since elementary 
school described his view on visualization as ability: 

“I think I was bom with it. I think it is something that could be developed, but I’m not sure if it can 
be taught. I don’t think everyone has that ability.” 

Donnie explained how he might be able to visualize something, but for some reason is unable to translate 
that to an external representation. Donnie’s struggles with generating a sketch may be influeneed in part by 
his limited field experience as he expressed his frustration in previous courses when professors required 
external representations from the students. Gerald not only believed that internal representation comes before 
external representation; he also believed that the ability to visualize can be developed. 

Bindings about Research Question 2 indicate that related work experience influences students’ 
representations. Students believe that real-world experience has an impact on their ability to create a mental 
image of how the system should look. Mel was an undergraduate student with two years of job site 
experience who uses hand drawn sketches on a regular basis as a communication tool with the site workers 
he supervises. He described how his related work experience has influenced his representations: 
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“I just draw on experience. I was always someone who understood how systems and things went 
together. Ever sinee I was a little kid... you know when I would see something work out or how something 
goes together I would expand on that and understand how other things might go with it..” 

The level of prior knowledge in a subject matter area seems to intervene with real-world experience 
influencing the creation of a visual representation. Participants reported that their prior knowledge from 
course work also helps them visualize a problem solution and serves as a complementing factor with their 
work experience. For example, Donnie recalled concepts learned from structural systems’ courses as 
contributing to his ability to visually recall correct examples of a system for the purpose of identifying errors 
with the task system, “My process for completing the task was based on knowing how it should work and 
where there would be a failure in the system due to the error.” 

The findings that prior knowledge and related work experience influence the ability to visualize appear to 
be consistent with situated cognition theory and the concept that knowledge is situated in the activity, 
context, and culture in which it is developed and used (Brown, Collins, & Duguid, 1989). Perhaps these 
findings mean a true relationship exists between subject matter knowledge and a situated experience. 


5. DISCUSSION AND IMPLICATIONS 

The results of the study indicate that the current model for instruction about the creation and use of 
representations should be revised. The current sequence of instruction, which consists of 1) instruction about 
tools and techniques to create an external representation (e.g., a sketch or computer generated drawing), 2) 
instruction about domain specific concepts, principles, and procedures, 3) individual real-world experience, 
and 4) instruction about internal representations, does not work well according to the findings. Tools and 
skills to use the tools do not influence participants’ external representation whereas internal representation 
does. Therefore, the instruction should be focused more on developing students’ internal representation, 
which would facilitate their external representation. In addition, it appears that students need to have 
experience applying their domain knowledge in a real-world setting prior to creating a representation. 
Consequently scaffolding tools need to be developed to support students’ ability to see a spatial-related 
problem and develop an internal representation without having to wait for real-world experience. Further 
research is needed to explore the relationships between internal and external representations, along with the 
relationship among each type of representation, self-efficacy, prior knowledge, and prior real-world 
experience. 
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ABSTRACT 

Innovative thinking skills are among the top characteristics that employers look for when hiring engineers. Universities 
are therefore charged with investigating and providing the type of learning environments that will foster the development 
of innovative thinking especially in large classes. This involves considering multiple factors such as the technology, 
teacher support, and learning task or educational affordances that are needed. This qualitative study examines 
undergraduate engineering students enrolled in large classes and their perspectives on educational affordances (i.e., 
technology, teacher support, and learning tasks) that they think will help them develop innovative thinking skills. 
Preliminary findings suggest that there are key factors in the educational environment that facilitate development of 
innovative thinking. Discussion will focus on effective practices and tools as well as suggestions for future research 
toward designing a model that presents effective pedagogical approaches to impact the development of innovative 
thinking skills among engineering undergraduates enrolled in large lecture classes. 
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1. INTRODUCTION 

The teaching and learning environment among undergraduate students of the 21st century continues to 
evolve. While the basic tenets of good teaching practice (Chickering and Gamson, 1989) remain relevant, 
factors such as students’ generational characteristics, class size, and technology usage present challenges as 
well as opportunities. In addition, universities must consider what type of learning environment will produce 
graduates that possess the skills needed to effectively perform in the working environment. 

Innovative thinking is considered a critical element in today’s economy. However, recent trends indicate 
that the U.S. is in danger of losing what has been considered its greatest competitive advantage. A study by 
the Information Technology and Innovation Foundation recently ranked the U.S. 40th out of 40 in the rate of 
change in innovation capacity over the past decade (Atkinson and Andes, 2009). Newsweek reported that for 
the first time since the 1950’s American creativity scores are falling (Bronson and Merryman, 2010). These 
and other findings implored a Wall Street Journal commentary to claim “The evidence is certainly mounting 
that we [the United States] face nothing short of an innovation crisis” (Lechleiter, 2010, para. 7). 

While previous research identifies cognitive processes that can be used to identify when innovative 
thinking is taking place, few formal educational curricula exist that describe a pedagogical method that can 
be used to develop innovative thinking among engineering students. Students tend to enter university with a 
propensity to be innovative, but after a rigid, structured program leave discouraged and less creative. This 
decline in innovative thinking is often attributed to the large lecture based classes that students are enrolled 
in, especially in the formative first and second years of their engineering degree program (Jamieson and 
Lohmann, 2009). 

Technology has allowed us to create enhanced learning experiences (Amelink and Scales, 2010; 
Chickering and Ehrmann, 1996). However, research indicates that comparing technology mediated 
environments with non-technology environments yields no significant differences (Clark, 1983; Lockee et 
al., 1999). 
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The question then becomes how does the use of technology facilitate learning of various skills in different 
contexts? In answering this question our attention is drawn to examining the affordances or the interaction 
between the features and characteristics of the learning environment and the learner (Kirschner, 2002). With 
this in mind this study seeks to answer the following research question: 

What are the educational affordances (i.e., technology, teacher support, and learning tasks) that facilitate 
the development of innovative thinking skills among undergraduate engineering students enrolled in large 
classes? 


2. CONCEPTUAL FRAMEWORK 

Based on research in engineering education (Pappas, 2009; Raviv, 2008; Raviv et ah, 2009; Raviv and Barbe, 
2010) and social constructivist learning theory (Jonassen et ah, 1993) we have identified seven (7) skills and 
related cognitive processes that can be used to identify development of innovative thinking among first and 
second year engineering students. These are knowledge acquisition (identifying new words and concepts, use 
of rehearsal strategies to memorize information.); scaling (organizing information and concepts so that they 
can be integrated into designing of new ideas and information.); elaboration (summarizing known 
information; ability to reframe content); critical thinking (application of previous knowledge to unknown, ill- 
defined and/or new situations; generation of new ideas); self-initiated exploration (ability to question ideas 
and information being presented); collaboration (ability to seek and entertain new ideas from peers and 
instructors; ability to utilize peers as a means to check new ideas and concepts); entrepreneurialism (use of 
team members to determine what creative ideas can become valuable innovations; effective presentation of 
new ideas to others). 

The term affordance was first used by ecological psychologist James Gibson in 1977 to describe the 
relationship/interaction between an animal/person and the characteristics of its environment, (Gibson, 1977). 
Since then the term has been adopted and adapted in other areas, and has developed context specific 
meanings in these domains (Norman, 1999). In the educational context what the learning environment 
‘affords’ will determine what learners gain. According to Kirschner (2002) the interplay between technology 
and learning is much more than the mere object and user, but is a “unique combination of the technological, 
the social, and the educational context” (p. 17). Thus he coined the term educational affordances which he 
defines as, “the relationships between the properties of an educational intervention and the characteristics of 
the learner that enable particular kinds of learning by him/her” (p. 19). 

A wealth of knowledge based in cognitive psychology identifies pedagogical approaches that can be used 
to facilitate more collaborative, active and engaging learning environments even in large lecture-based 
courses. This paradigm and the body of research associated with it suggests that instructional technology, if it 
is employed effectively, can serve as a means to engage students and promote active learning; facilitating 
educational environments that are related to the development of innovative thinking among undergraduates 
(Duncan and McKeachie 2005; Larwin and Larwin, 2011; Pintrich and Garcia, 1991). Foo et al. (2005) 
developed a “tripartite model” that consists of teacher support, teehnology and learning task. The model 
captures the dynamics of learning that takes place with technology and provides a framework for identifying 
the types of educational affordances that facilitate specific learning outcomes. They found that successful 
teachers focused on all elements of the model and those that were less successful focused on one or two 
elements of the model. Thus, more than the mere use of technology is required to successfully influence 
learning outcomes, rather the interplay of multiple factors need to be considered (Chickering and Ehrmann, 
1996; Clark, 1983; Foo et al., 2005; Salomon, 1990; Wijekumar et al., 2006). Our study seeks to examine 
those multiple factors and how they may influence the development of innovative thinking among 
undergraduates enrolled in large lecture classes. 


3. METHOD 

A qualitative methodology was used for this study. Three separate focus groups were held in order to try to 
better understand how undergraduate students enrolled in large lecture classes perceived educational 
affordances that might help develop innovative thinking skills. 
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Purposeful sampling was used to select the participants for the focus group. Three faculty members who 
taught large lecture classes and used very different pedagogical approaches were identified. Students enrolled 
in these courses were sent an email message inviting them to participate in a voluntary focus group and 
interested students responded and confirmed their attendance for the event. 

The protocol for all groups included questions that were designed to elicit information about students’ 
motivation for using technology or other learning tools, instructor’s pedagogy that they perceived helped 
them with innovative thinking and learning in general, and what tools they found helpful in the learning 
environment. All three focus groups lasted about one hour in length. The sessions were audio recorded and 
transcribed. In total 15 students attended the focus groups. 


4. ANALYSIS AND PRELIMINARY FINDINGS 

The focus groups were transcribed and are currently being analyzed. An a priori coding scheme, based on the 
educational affordances conceptual framework posed by Foo et al. (2005), along with the indicators (derived 
from social constructivist theory) of innovative thinking skills, is being used to identify themes that emerged 
in relation to how those affordances might facilitate development of innovative thinking among students. 

Preliminary findings suggest that there are key factors in the educational environment that facilitate 
development of innovative thinking. Some of these factors include effective faculty use of the technology to 
provide “just in time” learning, feedback, and demonstrations/illustrations. We plan on continuing to code 
and further analyze the data to refine the themes. 


5. CONCLUSION 

Kirschner (2002) advises that in order to truly accomplish learning goals, research should examine the 
learner’s perspective as only then will we be able to create the learning environments that ‘afford’ the types 
of behaviors we desire. The findings of this study will be very valuable as it will help in understanding how 
students perceive the learning support they get regarding innovative thinking as well as identify any gaps. 
The findings can be used to collate a list of best practices and tools, which can inform further research aimed 
at designing a model that presents effective pedagogical approaches to impact the development of innovative 
thinking skills among engineering undergraduates enrolled in large lecture classes. 
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ABSTRACT 

In an effort to keep up with the new generation of digital learners, educators are integrating multiple forms of technology 
into their teaching, including online learning game applications. The purpose of this study was to determine the degree to 
which internet-based learning game applications selected by preservice teachers were aligned with the Common Core 
Standards for Mathematics. Preservice teachers were trained using the SKATE Method and then chose online learning 
game apps that were easily accessible to school-age students. Using the Alignment Rating Scale, researchers determined 
how well the selected online learning game apps supported Common Core Standards for Mathematics. Findings indicated 
that only 66 percent of the online learning game apps supported the Common Core Standards for Mathematics, revealing 
a clear need for more curriculum alignment training for preservice teachers. 


KEYWORDS 

Learning game applications; mathematics curriculum alignment 


1. INTRODUCTION 

Curriculum alignment must be a central part in making instructional decisions in order to improve student 
success (McDonald and Van Der Horst, 2007; Squires, 2012). With the digital age upon us, more and more 
teachers are integrating technology into the classroom instruction, including the use of internet- based 
learning games. As a result, online learning game applications (apps) must be examined to determine how 
well they support classroom curriculum. This study served to identify the degree to which these apps 
reinforced Common Core Standards for Mathematics. Using the SKATE Method designed and developed by 
Jan Ray (Ray et al., 2011; Ray, 2006), preservice teachers identified, analyzed and selected internet-based 
learning game apps that were easily accessed by school-age children. The researchers then determined how 
many of these apps were actually in alignment with Common Core Standards for Mathematics. 


2. THE STUDY 
2.1 Background 

Education majors enrolled in the teacher preparation program at the university, take teaching methods 
courses related to mathematics, language arts, science, and social studies. During the mathematics method 
course, preservice teachers were introduced to the Common Core Standards for Mathematics, as well as to 
the Texas Essential Knowledge and Skills (TEKS) for Mathematics — state developed standards that 
determine classroom curriculum and instruction (Texas Education Agency, 2011). 
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In order to keep up with the new generation of digital learners whom preservice teachers would soon be 
facing in the classroom, they were introduced to the SKATE Method for aligning internet -based learning 
game applications with mathematics standards. The SKATE Method provided a framework that enabled 
preservice teachers to provide their future students with instructionally-sound, internet-based educational 
games, learning activities, instructional videos, and other resources (online supporting curriculum) that 
aligned with the written curriculum (state and national standards). 

The acronym SKATE represented the five steps preservice teachers took to successfully provide 
internet-based learning game applications for their future students. The structured steps were: 

1 . Start with STANDARDS . 

2. Eind KID-SAEE internet-based educational games, learning activities, instructional 
videos, and others resources that support standards-based instruction. 

3 . Check for curriculum ALIGNMENT. 

4. TEST for quality instructional design. 

5. Provide links to selected internet-based educational games, learning activities, 
instructional videos, and others resources for students through EDUBLOGs. 

Special Note: Edublogs, a shortened term for educational blogs, first emerged in the field in 2001. 
Edublogs have been used in educational settings for communications, instructional resources, collaborative 
tools, and showcases for students’ projects (Ray et ak, 2011; Ray, 2006, pp. 175-177). 

2.2 Purpose 

This study focused on the degree to which online learning game applications reinforced and were in 
alignment with Common Core Standards in Mathematics. Preservice teachers were trained using the SKATE 
Method to identify, test, and create an edublog with links to selected internet-based learning game 
applications. These apps were then rated to determine how well they supported Common Core Standards for 
Mathematics concepts. 

2.3 Research Question 

One research question was developed for this study: How well are internet-based learning game 
applications (identified, tested, and delivered by preservice teachers using the SKATE Method) aligned with 
Common Core Mathematics Standards? 

2.4 Participants 

Participants in this study were 35 preservice teachers enrolled in a mathematics methods course at a regional 
university in Texas. Of the 35 students, 32 (91.43 percent) were female and 3 (8.57 percent) were male. 
Subject ages ranged from 20 to 43 years of age, with 27 years of age being the average. Twenty-five (71 
percent) of the subjects were White. Seven (20 percent) of the students were Hispanic. Two subjects (6 
percent) were Asian American and one (3 percent) student did not respond to the question regarding 
ethnicity. 

2.5 Instrumentation 

An Alignment Rating Scale was designed to measure how well technological applications (apps) 
reinforced the curriculum taught in the Common Core Standards for Mathematics. This rating scale used the 
following numerical values and criteria. A score of 2 indicated that the technological application “clearly 
practiced” a specific math skill in the Common Core Standards for Mathematics. A score of 1 indicated that 
the technological application “marginally practiced” a specific skill in the Common Core Standards for 
Mathematics. A score of 0 indicated that the technological application “did not support” a specific math skill 
in the Common Core Standards for Mathematics. 
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2.6 Procedure 

Preservice teachers were introduced and guided through the SKATE Method for aligning the 
Common Core Standards for Mathematics with online learning game apps as follows: 

1 . S — Start with STANDARDS for Core Curriculum in Mathematics 

Preservice teachers became familiar with the Common Core Standards for Mathematics in their 
mathematics methods course. The student learning objectives were discussed and demonstrated in class. 
Preservice teachers then demonstrated the math skill to each other in small groups. Methods students 
discussed what new insights they gleaned and discussed those areas of each math concept that may be 
problematic to school-age students. 

2. K — Find KID-SAFE Online Learning Games 

Next, methods students performed searches using key words related to the standards. Apps were tested 
to determine if they were safe for students. Apps with inappropriate, questionable, or offensive content 
were excluded. As the methods students analyzed apps they discarded those that were unsuitable for children. 

3 . A — Check for Curriculum ALIGNMENT 

Apps were then analyzed to determine how well they aligned with the math curriculum. The 
preservice teachers documented each app and submitted them to the instructor for approval. 

4. T — TEST for Sound Instructional Design 

While previewing apps, preservice teachers quickly discovered that the online learning game applications 
varied greatly in their quality and degree of supporting quality standardized math curriculum. Participants 
played each app to determine if it was appropriate for use with school-age children and to what degree it 
supported or failed to support common core standards. Apps that displayed violence, or used negative 
responses to a student’s wrong answer were eliminated. 

5 . E — Create an EDUBLOG to Effectively Communicate Ratings with Classmates 

The simplicity of creating edublogs made it the perfect choice for effectively communicating app 
names and ratings among the preservice teachers and mathematics methods instructor. Using a free blog 
hosting site, preservice teachers were able to post links to the online learning game apps, along with the 
ratings each app received. Most preservice teachers were able to complete the entire task in less than two 
hours. 

Preservice teachers used the technology they readily accessed. This included ipods, iphones, ipads, droid 
cell phones, computers (both Apple computers and pc’s using Windows operating system). The Apple itunes 
store and Google Apps were the most commonly used for the methods students to locate and identify apps 
related to mathematics. Because these students will be certified to teach early childhood through sixth grade 
they tended to select apps for younger children. The math content most prevalent was whole number addition, 
subtraction, multiplication, and division. Most apps selected focused on fundamental elements of decimal 
numbers, counting coins, simple fraction concepts, and the number line. The apps were designed to reinforce 
mathematical concepts through repetition. The apps rated highest strongly supported the standardized math 
curriculum. The apps rated lowest minimally supported the Common Core Standards in Mathematics. 
Although the selected apps generally involved high interest activities, engaging readings, lively animation, 
motivational techniques, and frequent interaction, they were not rated on these criteria. This study focused on 
how well preservice teachers could identify internet -based learning games alignment with standardized math 
curriculum. 

2.7 Data Analysis 

The preservice teachers reported their apps in an edublog designed to communicate with all class 
members. All 175 apps were recorded into a common database with analysis. The Alignment Rating Scale 
was used by the instructor to evaluate the degree to which each app supported the common core mathematical 
standards. 
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2.8 Findings 

The findings for the research question revealed that of the 175 online learning game applications 
identified by preservice teachers, 66 percent were rated 2 (“clearly practiced” a specific math skill in the 
Common Core Standards for Mathematics), 30 percent were rated 1 (“marginally practiced” a specific math 
skill in the Common Core Standards for Mathematics), and 4 percent were rated 0 (“did not support” a 
specific math skill in the Common Core Standards for Mathematics). 


3. CONCLUSION 

This study revealed that 66 percent of online learning game apps identified by preservice teachers “clearly 
practiced” a specific math skill in the Common Core Standards for Mathematics and 30 percent of online 
learning game apps “marginally practiced” a specific math skill in the Common Core Standards for 
Mathematics. These findings indicated that only two-thirds of the identified online learning game apps were 
in alignment with standardized math curriculum. From these findings, it may be concluded that preservice 
teachers need more experience aligning supporting curriculum, such as online learning games applications, to 
the written curriculum (Common Core Standards for Mathematics). The researchers have two 
recommendations. The first recommendation is to work with preservice teachers until they can align 
curriculum at a rate of 90 percent or greater. Because of the low percentage of alignment (66 percent) 
achieved by preservice teachers in the area of mathematics, the researchers recommend that this study be 
duplicated in other content areas, such as, English/Language Arts, Social Studies, and Science to determine 
the degree to which preservice teachers are able to successfully select online learning game apps that support 
standardized curriculum. 
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ABSTRACT 

Does English proficiency affect participation in online discussion? This study polled 14 students from a postgraduate 
online course that require online discussion. The students are divided into groups according to their home language 
spoken and self-assessed English proficiency, and measure against their participation level in the required discussion 
forums. It is found that students who speak English at home posts more and longer, and students who rate themselves 
proficient in English post more and longer. However, literature suggests that factors affecting participation are multi- 
faceted. Although the current findings suggest a significant correlation between English proficiency and participation 
level, more work is required to understand the mechanism of the effects of language proficiency in online discussion 
participation. 
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1. INTRODUCTION 

Many educators recognize the merits of discussion to enhance learning experience (e.g., Andresen, 2009). In 
online learning environment, instructors may opt for asynchronous text-based forums to accommodate 
learners from different time zones. However, asynchronous online discussion still requires active 
participation. If most students do not participate, it is difficult to know whether any conclusion or consensus 
drawn from the participating minority is meaningful to everyone else. 

One obvious reason that affects participation is the language used in the forums. As most of Western 
colleges deliver their courses in English, English has to be the primary, if not the only, communication 
medium in online discussion forums. Students must be able to write good English to participate. The 
language barrier is not a trivial question. According to the Institute of International Education (2013), in 
2011/2012 the number of Chinese and Korean students in the US is over 260,000. In Canada, Chinese and 
Korean students comprise 41% of all international students, up to a number of more than 99,000 (Canadian 
Bureau for International Education, 2013). It is not clear how many of these students are taking courses that 
require online discussion. Nevertheless, if an instructor is expecting a significant number of East Asian 
students, it will be helpful to predict how they will participate in online discussion, given their higher chances 
of facing the language barrier. 

English proficiency is commonly assessed by standard tests, although the outcome of proficiency can be 
demonstrated in informal settings, including discussion forums. In other words, if a student writes well in 
discussion forums, he is proficient in English; if he is proficient in English, he writes well in the discussion 
forums. However, knowing about the proficiency of students after the discussion will not help educators to 
implement online forums. Instead of measuring English proficiency by collecting standard tests scores, this 
study select a couple of indicators that predict the proficiency, and by measuring the relationship between the 
indicators and participation level, it is hoped to help practitioners to implement discussion forums in online 
education environment. 
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2. RELATED STUDIES 

Research of participation in asynchronous online discussion is abundant, but with very diverse conclusions. It 
is suggested that effective participation are related to the design of the forums (Bassett, 2010), mediation and 
roles of instructors (Vlachopoulos & Cowan, 2010), types of tasks, intervention strategies (Andresen, 2009), 
learner characteristics (Yukselturk, 2010) and cognitive profile (Vercellone-Smith, Jablokow & Friedel, 
2012). 

Evidence on the effects of English proficiency, however, is inconclusive. Chen, Bennett and Maton 
(2008) reported a case study of two ethnic Chinese students in an Australian university. Although both 
participants are certified English teachers in mainland China, they reported frustration and fear when 
communicating with their instructor and fellow students through the discussion forums. Their perception led 
to unsatisfactory learning experience and outcomes. Similarly, Liu, Liu, Lee and Magjuka (2010) studied a 
group of Chinese students who took several online courses offered by a large Midwestern university in USA, 
and concluded that “language is still a dominant barrier for students who have English as a Second 
Language” (p. 186). In a qualitative study of the writings of non-native English speakers, Cronje (2009) 
concluded that the lack of presentation and communication skills is an essential problem. 

On the other hand, Bassett (2010) studied a group of students consisted of both English-as-a-second- 
language students and native English speaking students. He did not highlight the language proficiency as a 
prominent issue, but suggested that organization of the course is the most important factor contributing to 
effective online discussion. Earl and Cong (201 1) gave an account of eight Chinese students who took online 
courses offered by New Zealand universities. While the students agreed that the language barrier is 
important, they preferred the asynchronous discussion format which gave them the time to edit their posts. 
Campbell (2007) studied a mixed group of students and noticed that the English-as-a-second-language 
students participate less in online discussion. However, he addressed to the issue to their personality or 
shyness instead of their language proficiency. 


3. METHODOLOGY 

3.1 Background 

The target group of the study is the students who enrolled in the winter semester of a postgraduate project 
management course offered by a Canadian University. The project management course implemented two 
mandatory online discussion sessions. In each of the forums, students are required to initiate one post and 
respond to two other posts. The posts contribute up to 10% of the total grade. There is no bonus if students 
post more than required. 

3.2 Measurement 

It is important to operationalize and measure the two key concepts of the study - participation level and 
predicted English language proficiency. The measurement, however, is not without problem. More can be 
found in the Discussion section. 

Participation level measures how intense a student joins the discussion forums. The more posts and the 
more words they wrote, the more intense they participate. To quantify for participation level, I used two 
indicators - the number of posts students made and the number of words they wrote. Students’ posts and 
words are counted and matched to their response in the questionnaire. 

English proficiency is the ability to use a language to communicate with others. The indicators selected 
are the primary language spoken at home (home language), and a self-assessment of written English 
proficiency (self-assessed proficiency). Self-assessed proficiency is categorized into five categories: 
“Excellent”, “Good”, “Fair”, “Unsatisfactory” and “Poor”. In this study, the categories “Excellent” and 
“Good”, which reflects a certain level of confidence in using the English language, is classified as the 
proficient group. Respondents with “Fair”, “Unsatisfaetory” and “Poor” are classified as the non-proficient 
group. 
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3.3 Sampling and Data Collection 

There are a total of 15 students enrolled in the project management course. All students are invited to take 
part in the study when the course started. A questionnaire is developed to collect data. To avoid any possible 
effects of feeling coerced to give favorable answers, the questionnaire is released after all the grades are 
finalized. All 15 students returned the questionnaire. However, one student did not complete the survey and 
his response is removed from the study, and gives a total of 14 valid responses. 


4. HYPOTHESES 

Since there are two predictors of English proficiency (home language and self-assessed proficiency), and two 
indicators of participation level (number of posts, and number of words written), I have the following four 
hypotheses: 

(HI) Students who speak English at home write more posts. 

(H2) Students who speak English at home write longer posts. 

(H3) Students who rate themselves proficient in English write more posts. 

(H4) Students who rate themselves proficient in English write longer posts. 


5. RESULTS AND CONCLUSION 

Table 1 lists the average number of posts and words written by the students, breaking down by home 
language and self-assessed proficiency. The average of number of posts the student made is 6.07, and the 
average number for English speaking and non-English speaking groups are 12.89 and 7.17 respectively. All 
students write an average of 1990 words in both forums. The average number of words for English speaking 
and non-English speaking group is 2558 and 1139 respectively. 


Table 1. Average Number of Posts and Words by Home Language and Self-Assessed Proficiency 



Average number of posts 

Average number of words 

All respondents 

10.6 

1990 

Home language 

Speak English at home 

12.89 

2558 

Do not speak English at home 

7.17 

1139 

t-value 

1.73 (not significant) 

2.36 (significant) 

Self-assessed proficiency 

Excellent/ Good 

12.25 

2329 

Fair / Unsatisfactory / Poor 

4 

638 

t-value 

8.25 (significant) 

3.17 (significant) 


Two tailed t-tests are used to compare the differences between groups. As shown in Table 1, the t-value 
between English speaking and non-English group for number of posts (Hypothesis 1) is 1.73 with a level of 
confidence of 0.1066. The t-value for number of words (Hypothesis 2) is 2.36 with a level of significance of 
0.0347. The t-value between the self-assessed proficient group and the non-proficient group in terms of 
number of posts (Hypothesis 3) is 8.25 with a level of confidence of 0.0149. The t-value for number of words 
(Hypothesis 4) is 3.17 with a level of confidence of 0.0074. 

In summary, the data support all four of the hypotheses, i.e., it is found that students who speak English at 
home write more and longer posts, and students who rate themselves proficient in English write more and 
longer posts. The result for Hypothesis 1, however, is not statistical significant, while the results for 
Hypotheses 2, 3 and 4 are significant. 
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6. DISCUSSION 

6.1 Limitations 

The first limitation of the study is the sampling method. Although the data support all hypotheses, we cannot 
generalize the results to other group of students because of the non-probabilistic sampling procedure. 
Another limitation is the small sample size and the biased demographics of the samples. All respondents in 
the survey are male adults over the age of 21. All of them have some experience in online courses, but many 
of them do not participate in other discussion forums outside of their study. Both limitations may be rectified 
by larger scale study. For example, Thompson (2011) polled 287 students from four graduate business 
courses over a span of two years, with a conclusion of a multi -dimensional model of reasons of participation. 
A future study of similar scale with focus on language proficiency may yield more convincing results. 

A third limitation of the study is the measurement of participation. While we measure participation level 
in terms of number of posts and number of words written, both indicators only focus on the quantitative 
aspect of participation. Many studies employed coding schemes to measure the relevance and quality of the 
posts. For example, studies on Community on Inquiry usually involve encoding of transcripts to measure the 
three presences (Garrison & Arbaugh, 2007). However, if we hypothesize that language proficiency may 
have an impact on participation, we need to consider for any linguistic bias on the encoding scheme. 

Finally, the grouping of the self-assessed proficiency is arbitrary. While it is possible to use the rating of 
each response as cardinal data, it is not easy to justify that the distance between each of the response is the 
same. For sample, the magnitude between “Good” and “Excellent”, and between “Unsatisfactory” and 
“Poor” may not be the same. Interestingly, all the respondents in the “Fair/Unsatisfactory/Poor” group speak 
the same East Asian language at home. It may reflect a certain pattern of self-assessment of this country. An 
alternative to the classification scheme is to use Pearson’s coefficient to measurement the correlation. The 
coefficient between self-assessed proficiency and number of posts is 0.36, and number of words is 0.43 
respectively. Both show positive relationships. However, as mentioned in the Methodology section, it is hard 
to argue that the distance between Poor and Unsatisfactory, and between Excellent and Good, is equal. 

6.2 Factors Affecting Participation 

The current study concludes that there is a significant relationship between language proficiency and 
participation level. However, it is difficult to interpret the relationship given other evidences suggesting a 
complex web of factors affecting participation. To recapitulate from the Related Studies section, these factors 
include the demographics of learners (e.g., gender), pedagogy (e.g., mandatory or voluntary participation, 
active or inactive instructor intervention), social presence, and learner characteristics. The positive 
relationship between language proficiency and participation level may be but one of the many factors that 
affect the participation level. 

Given the various conclusions, there remains a question of how various factors interact with each other. 
Eor example, a latter study of the infamous Community of Inquiry model that the three elements - social, 
cognitive and teaching presence, may affect each other and progress over time (Akyol & Garrison, 2009). By 
the same token, is language proficiency modifies or interact with the three presences as well as other factors 
and affect participation? The question may be addressed through one of two approaches: (1), a large scale 
serial survey to collect enough data so that we can control significant variables, or, (2) an in-depth case study 
or focus group study that evaluate the process of participation of selected learners. In either case, further 
study is important to understand the nature of the role of language proficiency in participation level of online 
discussion for learning. 
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ABSTRACT 

Problem-based educational games are able to provide a fun and motivating environment for teaching and learning of 
certain subjects. However, most educational game models do not address the learning elements of problem-based 
educational games. This study aims to synthesize and to propose the important elements to facilitate the learning process 
and problem solving skill in game-based learning from selected education game models and a problem-based learning 
model. The results showed two critical components were 1) pedagogy and 2) design embedded in game-based learning 
and three significant characteristics of a problem-based educational game were 1) authentic problems, 2) student-centered 
activity, and 3) a facilitator. This paper focuses on proposing and discussing components that leverage the pedagogical 
aspects in designing a problem-based educational game environment. Moreover, other developers can use this pattern to 
facilitate to he student-centered learning environment. 
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1. INTRODUCTION 

The digital games concept has been used in education since the end of the 20th century because games have 
the potential to promote the student’ skills through problem solving. This paradigm, known as game-based 
learning, was one in which Conati (2002) identified games utilization as a medium for conveying the learning 
content. Gee (2003) defined educational games as games that emphasize learning and designing to teach 
humans about a specific subject, and to teach them a skill. Game-based learning is similar to problem-based 
learning, that specific problem scenarios are placed within a play framework (Barrows & Tamblyn, 1980). 
Problem-based learning can provide a student -centered learning approach (Walker and Shelton, 2008). 

As Barab et al. (2005) described, educational digital games could draw students’ attention and allow them 
to develop their experience and cognition through the games. Instructional designers and game developers 
should consider more the pedagogical basis of a game. Real learning does happen in games, and the learning 
engaged by gamers shares many attributes with the pedagogy of problem-based learning. Players must solve 
problems to progress through the game. Challenging, unpredictable, and competitive digital games are the 
drivers of game playing (Squire et al., 2004). A digital game is a form of entertainment which has 
characteristics such as goal, rule, competition, challenge, fantasy, safety, interaction, problem-solving, results 
and feedback, as well as storyline, (Alessi & Trollip, 2001; Prensky, 2001). Buchanan (2004) insisted that 
digital games should be designed to support learning of cognitive, affective, and psychomotor skills abilities, 
problem-solving thinking skills and promote creative exploration. However, to design a game and know how 
to make the placement of learning content in order to achieve the instructional goal and let the players 
achieve knowledge is an important issue. Educators attempt to develop a complex simulated environment to 
nurture learners’ problem-solving ability because games can be offer the situations to explore, reason, decide, 
and apply related abilities to similar situations in daily life. 

This study aimed to propose and to synthesize important elements to facilitate the learning process and 
development of problem solving skills in game-based learning from the selected education game models and 
a problem-based learning model. 
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This paper is organized as follows: Section 1 provides the background of this case study; Section 2 
illustrates some models for problem-based educational games; Section 3 presents some ideas and elements 
that a problem-based educational game should contain and the last section of this paper includes future 
directions following this review. 


2. PROBLEM-BASED EDUCATIONAL GAME MODELS 

This section discusses four selected models proposed by Amory and Seagram (Game Object Model: GMO), 
Song and Zhang (EMF: A Model for Education game design), Simons and Ertmer (Model of problem-based 
learning) and Kiili (A Problem-based gaming model) which fall into the category of problem-based 
educational games. 

The models were analyzed based on 2 major components: (1) Pedagogical (exploration, challenges, 
engagement, goals, critical thinking, competition, practice, student-centered learning, motivation, and task) 
(2) Game Design (storyline, outcome, interaction, collaboration, facilitator, feedback, coaching and 
reflection, and reflection & debriefing) 

Table 1. Summary Elements of Problem-Based Learning Models 


Component 

Pedagogical 

Design 

\ Elements 

Models \ 

Exploration 

Challenges 

Engagement 

Goals 

Critical thinking 

Competition 

Practice 

Student-centered 

learning 

Motivation 

Task 

Storyline 

Outcome 

Interaction 

Collaboration 

Eacilitator 

Eeedback 

Coaching & 
Scaffolding 

Reflection & 
Debriefing 

Amory and 
Seagram (2003) 

• 

• 

• 

• 

• 

• 

• 




• 


• 



• 



Song and 
Zhang (2008) 


• 

• 

• 




• 

• 




• 



• 



Simons and 
Ertmer (2005) 


• 

• 

• 

• 



• 

• 


• 

• 

• 

• 

• 

• 

• 

• 

Kiili (2007) 


• 

• 

• 

• 



• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 


Table 1 illustrates a summary matrix of the game design element for the models. All the models 
emphasize greatly the interaction, challenges, goals, engagement and feedback elements. However, only 
three models include engagement, critical thinking, motivation. Student-centered learning, and storyline. 
Outcome, collaboration, facilitator, coaching & scaffolding, and reflection & debriefing are some common 
elements shared in different combination of two models. 

2.1 Game Object Model 

Amory and Seagram (2003) presented a model called “Game Object Model” (GOM) which combines 
education theory and game design. This model consists of both pedagogical dimensions and game elements. 
Pedagogical elements are play, exploration, challenges, engagement, goal formation, goal competition, 
critical thinking, discovery, competition and practice. On the other hand, games elements that are interaction, 
storyline, feedback, fun, graphics, sound and technology. 

GOM is the model that game designer should use when developing an educational game. It contains the 
pedagogical elements and game elements that can create a learning environment. 
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2.2 EFM: A Model for Education Game Design 

Song and Zhang (2008) implemented an educational game design model for educational game design through 
describing motivation, flow experience and effective learning environment. This model provides some design 
guidelines for researchers and developers of educational games. The model includes: 

1. Effective learning environment: It must provide players with the seven basic requirements, which 
include goals, challenge, motivation, engagement, appropriate tools, avoiding distractions, feedback, and 
challenge. 

2. Motivation: The four essential strategy components for stimulating motivation are relevance, 
confidence, satisfaction and attention, which relate to the four elements which are goal, challenge, feedback 
and interest. 

3. Flow experience: The nine dimensions of the flow experience also include goals, feedback, challenge, 
concentration, control, awareness, experience, time, and self-consciousness. 

2.3 Model of Problem-Based Learning 

This model has its foundation in the theories of student -centered learning. Simons and Ertmer (2005) pointed 
out that problem-based learning presents several challenges to the learner. This model ranges from limited 
implementation, which present a question with the teacher or facilitator to the learner. The elements of 
problem-based learning model are engagement, motivation, collaboration, critical thinking, student-centered 
learning, storyline, outcome, goals, facilitator, feedback, challenges, coaching & scaffolding, reflection & 
debriefing and interaction. 

2.4 A Problem-based Games Model 

Kiili (2007) proposed a model representing the steps for problem-based game design. The model is founded 
on the principles of problem-based learning. This model assumes out that there are 15 factors which must be 
viewed in a problem-based game i.e. engagement, challenges, goals, critical thinking, student-centered 
learning, motivation, authentic learning tasks, outcome, interaction, feedback, collaboration, storyline, 
coaching & scaffolding, and reflection & debriefing and facilitator. All the factors are closely related to the 
learners. 

The factors covered in this model are important in designing good problem-based games. This aim of this 
model is to describe a general learning process in an educational game. 


3. GAME DESIGN: ELEMENT EOR PROBLEM-BASED EDUCATIONAL 
GAME 

An appropriate problem-based educational game should be developed based on some criteria which fulfill the 
requirement of the target learners. A part form student -centered learning, game design is another component 
that game designers should consider when developing problem-based educational games. Design of a 
problem-based educational game might affect the learning process of learners when playing the game. Thus, 
the design of the game is very important. From the review of the models, we suggest that the problem based 
educational games should have three important characteristics, i.e. authentic problem, student -centered 
activities, and facilitator. 

3.1 Authentic Problems 

Authentic problems are the central focus of instructional activities in problem based educational games. 
Authentic problems in educational games are designed, selected and sequenced in the way that guide student 
learning. 
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An example of presenting authentic problems would be that the problem has multiple solution paths, the 
problem is unsolved and ill-structured, the environment is abstracted as a design decision to support the 
authentic nature of task, and learning is followed up with opportunities for practice in more realistic settings. 

3.2 Student-centered Activity 

Problem-based educational games are a learner centered approach to education (Walker and Shelton, 2011) 
Specifically, the player take on responsibility for examining a problem in a game and then determining what 
they know and what they need to learn in order to solve it. Problem based educational games still have a 
responsibility for defining learning objectives, but these are mostly kept from players who generate their own 
learning objectives in response to a given problem. The next step is locating appropriate resources to allow 
investigation of areas of personal interest. Problems in a game needed to be presented prior to any pursuit of 
knowledge. As a final step, players are responsible for self-assessment and peer assessed problem solving 
performance. The examples of student-centered activity in problem-based educational game are activities that 
a player uses to generate objectives from given information, locate and pursue resources in a form that will 
assist in problem resolution, and engage in self and peer assessment of problem solving performance. 

3.3 A Facilitator 

Problem-based learning instructors act primarily as facilitators, a role shift that support students throughout 
the problem-solving process. So facilitators in problem-based educational game are in a Multi-User Virtual 
Environment. Within a Multi-User Virtual Environment, learner action is mediated by a progression of 
analysis of tasks within the environment. Progress is monitored by an instructor who provides coaching and 
guidance. Eacilitators in problem-based educational games are designed to be an engaging way to improve 
educational outcomes. The examples of facilitators enable multiple participants to access virtual configured 
architectures, interact with digital artifacts, and communicate with other participants. 

Design of the Problem based on educational games might affect the learning process of learning when 
learners are playing the game. Thus, the design of the Problem based educational games is important. Prom 
the review, we suggest that the Problem based educational games should contain 19 subcomponents i.e. rule, 
goals and objectives, storyline, feedback, competition challenge, fantasy, control, curiosity, feedback, 
outcome, fun, interaction. Coaching & Scaffolding, Reflection & Debriefing, Authentic problem and 
Pacilitator in Pigure 1 
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4. CONCLUSION 

Educational games may offer a viable strategy for developing student’s problem solving skill and so 
developing a good educational game for players is very important. Several models have been proposed by 
researchers in field of game design. However, the development of a game to be used for learning in a 
difference form requires game designers to look in future into the learners and learning content 
pedagogically. 

This paper has identified the essential components based on existing models. Our thorough study of the 
existing models showed two critical components were pedagogy and design embedded in game-based 
learning and three significant characteristics of problem-based educational games were authentic problems, 
student-centered activity, and a facilitator. Moreover, all developers can use this pattern to facilitate to be 
student-centered learning environment. 
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ABSTRACT 

Teaching and learning in Web-based courses has become a global phenomenon. Educators are grappling with merging 
cognition and technology to offer students quality, relevant online courses. The development of social presence in the 
online environment is of paramount importance and requires individuals to engage in meaningful interactions about, and 
with, the course content. It is the richness of examining various perspectives and dialogue that stimulates critical thinking, 
reflection, analysis and synthesis of theory that further enables students to apply the newly learned theory to practice. The 
purpose of this paper is to present an adapted version of the Challenge-Based Learning ™ (CBL) method (Bransford, 
2012) and its utility in an online emergency preparedness and disaster planning course. The six key sections of the CBL 
cycle will be discussed and examples of their practical application in an online environment will be presented. CBL 
Cycles in this course are learner-centered and maximize the use of learner-generated content thus enhancing the value of 
active participation and engagement, which is at the very core of successful online education. Student and faculty 
perspectives related to the CBL method of learning will be discussed and lessons learned from the use of this adapted 
teaching pedagogy within the online environment will be shared. 


KEYWORDS 

Student-centered, discovery-based online learning 


1. INTRODUCTION 

“I hear and I forget. I see and I remember. I do and I understand” (Bradford, 2012, p. 1). The Challenge- 
Based Learning (CBL) Cycle is based upon discovery-based learning and requires the student to take control 
and be responsible for their own learning. This presentation will review the premise upon which the CBL 
Cycle was adapted for use in the online environment and provide examples from each section of the Cycle to 
walk the participants through the lived experience from the perspective of the student to illuminate the union 
of cognition and technology. The Cycles require a high degree of interaction with the content and with the 
students’ classmates. The extent to which students in online education feel socially connected is frequently 
cited as a key factor in the perceived success of online courses (Slagter van Tryon & Bishop, 2009) and 
immersion in scholarly discussion is key to achieving learning outcomes related to emergency preparedness 
and disaster management. It is the students and their collective wisdom throughout each of the CBL Cycles 
that make positive online learning environments productive, not content (Kehrwald, 2008). 

CBL encourages the student to engage with the content on an intellectual and emotional level initially 
through visual stimulation. The Think About Questions require the students to ground themselves in a 
particular central role in relation to the newly learned theory and leads them to critically think about the 
scenario and reflect upon their feelings, inspiring students to document their initial thoughts on a personal 
and professional level; in this course, it would be the students initial thoughts related to the disaster depicted 
in the Scenario Clip. A virtual discovery-based learning environment is created as students engage in 
research and scholarly discussion with their peers for a one week period. The research that is completed does 
not offer the student the “answers” to the questions but rather provide various perspectives related to the 
discussion topic. Students return to their journal at the end of the cycle to record their Final Thoughts; 
evidence of reflection, analysis and synthesis of the information is expected. 
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2. CHALLENGE BASED LEARNING 


2.1 Challenge-Based Learning Cycle in an Online Learning Environment 

Drawing heavily from constructivist learning theory (Legg et ah, 2009), discovery learning is inquiry-based 
and engages the student in the content that is to be learned or applied. Discovery-based learning takes place 
in problem solving situations where the learner draws on his or her own past experience and existing 
knowledge to discover facts and inferences to apply to new situations (Alfieri et ak, 2010). This type of 
learning requires the student to assimilate and accommodate new learning. Students interact with the research 
resources by thinking about what they are learning and learning from each other to explore questions and 
controversies. Students are more apt to engage in the material and remember knowledge that they have 
discovered on their own. 

The original CBL cycle had four sections, namely, challenge, initial thoughts, resources and final 
thoughts. Adaptations were made to incorporate the CBL Cycle in an e-learning platform and to emphasize 
social interaction and dialogue. The two additions to the Cycle included adding a private journal for students 
to reflect upon the Think About Questions and altering the Resource section for two learning activities, 
namely, a Research Room and a Discussion Room. The adapted CBL Cycle includes a scenario clip (the 
challenge), think about questions, initial thoughts, research room, discussion forum and final thoughts. In this 
online course the scenario clips depict a realistic type of disaster to spark the students’ interest and engage 
them in the situation. Immediately after viewing the scenario, students are asked to record their initial 
thoughts, feelings, reactions, or questions that arose while viewing the clip. The addition to the cycle of a 
journal was done to enable students to reflect upon the situation that they have watched and to privately, and 
openly, document their feelings and reactions. The journal is a private area on the e-learning platform that is 
only visible to the student and the professor. At this point in the cycle, students are using what information 
they already had prior to viewing the scenario as well as previous problem solving skills. This section of the 
CBL Cycle is to entice students to think about what they would do in the situation and begin to formulate 
their thinking about this type of disaster. 

Six to eight Think About questions are posted with each different type of disaster to encourage students to 
begin to think about the what, why, who and how of the situation. The goal is to have the student think of 
themselves as the nurse in the scenario and what would they do in this particular situation. The focus is on 
students acquiring a “walk in the nurses’ shoes” feeling and personal and professional engagement with the 
content. 

Next the students enter the online research room. At the beginning of the Cycle there is one professor 
posted resource related to the scenario and two student posted resources. The research room remains ‘open’ 
all week to enable group members to post additional learning resources that they find particularly helpful and 
relevant. The online environment is conducive to this type of discovery-based learning structure. It 
encourages active engagement, promotes motivation, autonomy, independence and responsibility for one’s 
own learning. In addition, the asynchronous nature of the CBL Cycles in this course allows time for students 
to locate and share additional resources within their learning groups. The research room does not contain the 
answers as to what the nurse should do in that particular disaster but they present differing perspectives- 
which in this course is the role of the nurse in emergency preparedness and disaster management. 

After reading the research students enter the discussion forum where they participate in a week long 
asynchronous discussion with their peers in small groups. Students are required to post focus questions to 
help guide and stimulate each other’s thinking. In addition, each week there is a student in the role of 
Research Leader and Moderator for that particular CBL Cycle. The role of the professor is to monitor all of 
the small group discussions and to intervene only if there is inaccurate information represented or 
interpersonal issues related to group dynamics arise that hamper communication within the group. 
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At the end of the Cycle, students return to their journal and record their final thoughts. They are able to 
review their initial thoughts and journal about how their thoughts have changed over the course of the week 
after having been exposed to current research about the topic and engaging in scholarly discussions with their 
peers. The goal of the journal is to enable the students to write about what they have critically thought about 
and discussed throughout the Cycle, providing evidence of knowledge application, analysis and synthesis. 
Once the Cycle is completed all of the discussion boards are locked so that no further postings can take place 
and are shared among all of the groups, further adding to the students learning in relation to the various 
perspectives from their classmates. Figure 1 provides a graphical depiction of the CBL Cycle used within the 
online platform. 



Eigure 1. Graphical depiction of an adapted Challenge-Based Learning Cycle for the online learning environment. 

2.1.1 Thoughts from Learners 

At the end of this course students were given an evaluation form to complete in relation to the CBL Cycles. 
At the time of this writing, the evaluative data from the learners is not yet available but will be prior to the 
conference to enable this information to be included in the oral presentation. Anecdotally, students are 
expressing positive comments about the CBL Cycles and are embracing their research and leadership roles 
beyond the expectations of the professor. The goal of the course evaluation is to collect data that will inform 
future course offerings to improve the experience and to investigate students’ impressions and experiences 
while engaged in this type of learning online. Evaluative data to be presented includes the students overall 
impression of the course, their thoughts related to the CBL Cycles, what could be done to improve learning 
while using the CBL Cycles, the level of engagement in relation to other online courses they have taken, and 
an open ended question asking for any additional thoughts or suggestions to improve the use of CBL Cycles. 
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2.1.2 Lessons Learned and Suggested Alterations 

Depending on the compilation and analysis of the evaluative date, lessons learned and suggested revisions to 
the CBL Cycles will be presented. 


3. CONCLUSION 

The CBL Cycles are based upon a constructivist paradigm that revolves around discovery-based learning. 
CBL Cycles used in this online environment have stimulated student thinking, engaged students in learning 
new content and competencies, and empowered them to take control of their own learning. Student 
engagement with the material in this course has far exceeded professor expectations. Learning in this manner 
affords the student with a memorable and effective learning experience, while maintaining all of the cited 
advantages of online education. Uniting cognition and technology by way of using CBL Cycles in an online 
learning environment has proven to be an effective and exciting union that holds great promise for long-term 
retention and application of information. Thoughts from the learners were presented in addition to lessons 
learned from the initial use of CBL Cycles in this course were shared. Suggested revisions for improvement 
in the CBL Cycles will be presented as well as consideration for use of CBL Cycles in other courses. 
Students were not just memorizing and regurgitating knowledge while engaged in the CBL Cycles; they were 
doing, and when we do, as an active learner, we remember! 


ACKNOWLEDGEMENT 

Thank you to Dr. John Bransford for permitting me to adapt and use the Challenge Based Learning Cycle 
method. A grateful thank you to Dr. Betsy Weiner for permission to use the scenario clips to initiate 
discussions of various disaster scenarios. 


REFERENCES 

Alfieri, L., Brooks, P., & Aldrich, N. (2010). Does discovery-based instruction enhance learning? Journal of Educational 
Psychology, 103(1), 1-18. 

Bradford, J. (2012). Challenge-Based Learning, (p. 1-6 online). Retrieved from 
http://littleplanetlearning.wordpress.eom/2012/02/01/challenge-based-leaming/ 

Kehrwald, B. (2008). Understanding social presence in text-based online learning environments. Distance Education 
29(1), 89-106. 

Legg, T., Adelman, D., Mueller, D., & Levitt, C. (2009). Constmetivist strategies in online distance education in nursing. 
Journal of Nursing Education, 48(2), 64-9. 

Slagter van Tryon, P., & Bishop, M.J. (2009). Theroetical foundations for enhancing social connectedness in online 
learning environments. Distance Education 30(3), 291-315. 


344 



lADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2013) 


STUDENT-DRIVEN CLASSROOM TECHNOLOGIES: 
TRANSMEDIA NAVIGATION AND TRANEORMATIVE 

COMMUNICATIONS 


Leila A. Mills, Gerald A. Knezek and Jenny S. Wakefield 

University of North Texas, 3940 Elm Street, Denton, TX 76207 


ABSTRACT 

This research paper explores middle school student attitudes towards learning with technology and proposes a 
design-hased approach to formulating instruction that includes innovative classroom technology use with computers and 
communications technologies placed in the hands of students. The intent of this research is to advance practice and theory 
in innovative classroom use of information and communications technology (ICT), going beyond the implementation of 
school technologies for delivery of lessons and data processing. The focus of the recommended design-based approach is 
teaching and learning that provides opportunities for student-driven ICT use in the classroom, including transmedia 
navigation for classroom activities that encourage students to think, interact with instructional content, and engage in 
transformative communications. 


KEYWORDS 

Design-hased research, transmedia navigation, transformative communications 


1. INTRODUCTION 

Research indicates that all too often school technology is being used to support new options for delivery of 
traditional instruction rather than technology integration for teaching and learning (Bauer, Kenton; 2005). 
Cuban (2001) stated that the instruction in schools has changed very little in the last 20, 50, or even 100 
years. Little progress has been made in school technology as a vehicle to deliver innovation for classroom 
learning activities (Halverson and Smith, 2009). Renowned technology expert, computer scientist, and 
educator Seymour Papert envisioned the transformative power of a new kind of system for education, one 
comprised of a student and a computer (Papert and Harel, 1991). Papert and Harel (1991) recognized that 
technology-related change in education would face an uphill battle. The dream of Papert has yet to be 
actualized in the majority of schools. That is, “although technology enthusiasts expected a revolution in 
technologies for school learners, what schools experienced was a revolution in technologies for measuring 
and guiding learning” (Halverson and Smith, 2009: 53). 

This research is based on a sample of middle school students and their attitudes toward learning with 
technology and suggests a flexible design-based approach to the design, implementation, evaluation, and 
extension of successful strategies for student-driven use of classroom technologies. A focus on classroom 
discourse, transformative communications, and transmedia navigation within a design-based research 
framework are recommended as methods to extend knowledge and practice of successful, innovative 
classroom use of ICT. 
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2. CONCEPTUAL RATIONALE 

2.1 A Design-Based Methodology 

Design-based research is a systematic yet flexible methodology set in place to improve practice through 
iterative analysis, design, development, implementation, and re-redesign. This research and practice 
paradigm requires collaboration among researchers and practitioners and can lead to improved practice and 
revised theories (Van den Akker et ah, 2006; Wang and Hannafm, 2005). Brown (1992) warned that 
examination of separate processes, as isolated variables within laboratory or other impoverished contexts, 
will provide incomplete pictures. A design-based methodology, however, allows researchers to incorporate 
theories and approaches to the classroom practice with continual re-evaluation, allowing for testing and 
validation of theories through a series of adjustments in application. Each adjustment can be considered a 
form of experiment that allows for generation of new theories for learning in naturalistic contexts (Brown, 
1992; Collins, 1992). 

2.2 Transformative Communications 

Slow if gradual progress is being made towards acceptance of Dewey’s assertion, offered nearly 100 years 
ago, that knowledge and common understanding cannot be passed physically from person to person as can 
bricks, nor can they be shared among persons as could a sliced pie (Dewey, 1985). Dewey, father of 
American pragmatism, believed it is fair to claim that any social arrangement that is vitally social, or vitally 
shared, is educative to those who participate in it, and further, that social life is identical to communication, 
the central process of education. Sharpies (2005) identified a need for a conceptual framework that 
recognizes the essential role of communication for learning in the mobile age. Sherry and Wilson (1997) 
conceptualized a transformative, dynamic, two-way system of communication for education within the Web 
environment that combines elements of transmission and ritualistic views of communication. 

Learning and teaching as communicative actions theory (ETC A) (Warren et ah, 2010) supports 
instructional design for learning and teaching through classroom discourse within technology-enhanced 
environments of the 21st century. ETC A discourse allows shared expression of identity and meaning making 
within a learning and teaching context that comprise four essential communicative actions defined as 
normative, strategic, constative, and dramaturgical (Wakefield, Warren & Alsobrook, 2011). The 
dramaturgical action (expressions of subjective understanding in the objective world), in particular, supports 
dynamic new models of classroom instruction that extend instruction beyond traditional classroom walls. 
Dramaturgical communicative actions are the results of individual understanding from interactions with 
content and ideas. The LTCA framework for learning has been used in Web environments utilizing tools 
such as Twitter (Wakefield, et al., 2011), blogging (Warren and Wakefield, 2012), in role play within virtual 
worlds (Wakefield et al., 2012), and in transmedia storytelling within the educational setting (Warren et al., 
2013). 

2.3 Information Technologies in Education 

Erom the rise of the motion picture in the 1920’s, to the appearance of the computer in the mid-1970s, 
educators have been intrigued with the potential of technology to aid in the transformation of education and 
improvement of student learning (Hew and Brush, 2007). Visionaries such as Paper! and Harel (1991), 
predicted computers would be more than powerful classroom tools that would allow learners to construct and 
test complex hypotheses. Engelbart (2001) studied systems with the potential to improve effectiveness of 
individuals by connecting tools and methods suited to the capabilities of the person to the problem at hand. 

School technology implementations typically have not been in the direction of student use of technology 
tools as media for interaction with classroom content within the school (Halverson and Smith, 2009). While it 
has been shown that appropriate application of information technology can, in fact, enhance student learning 
(Voogt and Knezek, 2008), the promise of personal computers as tools with power to support new systems 
connecting learners, instructors, and digital information for learning interaction and knowledge construction 
continues to be mostly unfulfilled (Halverson and Smith, 2009). 
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2.4 Classroom Activities: Transmedia and Learning 

To learn, Ritchhart et al. (2011) noted, we must think and engage with content. There is, however, concern 
among educators that trends toward accountability-driven instruction have limited learning opportunities for 
students (Ravitch, 2010), decreasing instances of critical discourse, problem-solving, and collaboration 
within the classroom. One option for the introduction of engaging instruction in the 21*' century classroom is 
interactive instruction implementing transmedia. Transmedia navigation, as noted by Jenkins et al. (2006: 4), 
is the ability to conduct research and follow topics, stories, and ideas “across multiple modalities” or media. 
Transmedia navigation can serve an important component of a technology-mediated learning paradigm 
known as the New Media Literacy (NML) framework (Jenkins, et al., 2006). Correctly designed, a 
transmedia lesson that offers student a choice from a variety of media, to include ICT tools, for interaction 
with contents can function as a blend of student-directed and instructor-mediated research, discourse, and 
expression that can provoke student thinking. A very simple implementation of transmedia navigation for 
middle school students might allow students to read in class using paperback books, the Kindle, or iPads; and 
topic related discussion might include classroom discussion and posts to shared blogs (Warren, et al., 2013). 
As noted by Wakefield et al. (2013:1612), “Learning through digital tools, actively seeking content, weaving 
together storyline, sharing, communicating, and expressing individual understanding - arguing, defending, 
and critiquing - may provide a way for cognitive learning in a community” and will additionally provide 
opportunities for development of NML skills for collaboration, problem-solving, and knowledge 
construction. 


3. STUDENT ATTITUDES TOWARDS LEARNING WITH 
TECHNOLOGY 

Student attitudes can be measured with reliable, validated survey instruments such as the Computer Attitude 
Questionnaire (CAQ) (Mills, 2013) which includes five parts that build subscales to gauge: Computer 
Attitudes (Comfort with Computers and Learning with Computers), Empathy, Creative Tendencies, 
Motivation, Study Habits, Empathy, Self Concept, and Attitudes toward School. The CAQ has foundations in 
an instrument designed for the youngest school-age children: the Young Children’s Computer Inventory 
(YCCI). The research and development of the CAQ was funded by the Eulbright Eoundation of Washington 
DC, the Japan Society for the Promotion of Science, and the Texas Center for Educational Technology. The 
CAQ was formalized as a validated measurement tool in 1995 and was extensively used in research studies 
(Knezek and Christensen, 1995, 1996, 2000) before being released for public use in 2000. It was then 
revalidated in 2011 (Mills et al. 2011). The CAQ is comprised of 52 Likert-type with a 5-point response 
ranging from 1 = strongly disagree to 5 = strongly agree. Cronbach’s Alpha internal consistency reliability 
for the CAQ subscale Learning with Computers {alpha = .83) and Creative Tendencies {alpha =. 88) were 
found to be very good, according to guidelines by DeVellis (1991). One new prototype item was added to the 
Learning with Computers subscale for the purpose of this study: The more often I use a computer at school 
the more I enjoy school. 

Selected items from the Student Attitudes Inventory (SAI) where also examined during this study. The 
SAI instrument gauges students’ school-related dispositions. The SAI was developed for studies at the 
Hawaii State Department of Education (Dunn-Rankin, et al., 1971). This instrument and its subscales 
continue to demonstrate strong internal consistency and reliability today (Mills, et al., 2013). 


4. STUDY PARTICIPANTS 

Survey subjects were students enrolled in one of two middle schools in a public school district in north 
Texas. The sixty-three (63) students completed a survey battery to report their attitudes towards school, 
technology, and learning during the spring semester of 2012. Survey participants were 48% boys (n = 30) and 
52% girls (n = 33). 
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5. FINDINGS 

Student survey data, as reported on the CAQ and SAI, were analyzed for Pearson’s product-moment 
correlations in order to examine students’ attitudes towards school, ICT, and learning. 

Significant positive correlations were identified between student perceptions that computers give me 
opportunites to learn many new things and: 

1) the more often the teacher uses computers the more I will enjoy school (r = .33, p = .009) 

2) the more often I use a computer at school the more I enjoy school (r = .46 p < .0005) 

3) I find learning new things interesting (r = .40, p = .001) 

These reponses indicate that students who associate computers with learning opportunities feel that 
computer use at school makes school more enjoyable. 

Significant positive correlations were also identified between students’ perception that I would work 
harder if I could use computers more often and; 

1) Computers give me opportunities to learn many new things (r = .50, p < .0005) 

2) I find learning new things interesting (r = .36, p = .004) 

3) the more often the teacher uses computers the more I will enjoy school (r = .40, p = .001) 

4) the more often I use a computer at school the more I enjoy school (r = .48 p < .0005) 

These responses indicate that there is a relationship between feeling motivated to work harder when using 
computers and percieved opportunity for and interest in learning new things. These data also indicate a 
relationship between feeling motivated to work harder when using computers and feeling that school 
computer use makes school more enjoyable. 

A positive significant relationship was identified between the items I enjoy books, newspapers, magazines 
and I really like school (r = .36 p =.004). These responses indictate that students who enjoy traditional 
learning media, such as books, tend to like school. A negative significant relationship was found between I 
enjoy books, newspapers, magazines and the more often I use a computer at school I the more I enjoy school 
(r = -.28 p =.024). This negative correlation indicates that students with high preference for learning with 
traditional learning media do not feel that school ICT makes school more enjoyable. This implies that 
students who do not enjoy traditional media, such as books, tend to not like school. 

Additional findings were that subscale scores from CAQ indicated students’ preferences for Learning 
with Computers align with perceptions of having Creative Tendencies (r = .51, p < .0005), indicating a 
significant trend between students’ positive perceptions of learning with computers and perceptions of 
tending to have creative tendencies. The magnitudes of Pearson product-moment correlations reported would 
be considered moderate and medium effect sizes, according to guidelines by Cohen (1988). 


6. DISCUSSION AND CONCLUSIONS 

The promise of personal computers and ICT as a powerful force for school change and improved student 
learning remains largely unfulfilled, yet the advent of personal computers and Internet based ICT tools are 
forcing a re-conception of teaching and learning. This study of student attitudes towards learning with 
technology reveals that students who report a dislike of traditional learning media, such as books, 
newspapers, and magazines also report liking to learn new things and being motivated to learn when 
computers are used in learning and teaching. The authors suggest that the promise of enhanced student 
learning with technology may be fulfilled if classroom instruction is designed via the design-based research 
framework, flexibly adjusting the design and composition of lessons based on the dynamics of classroom 
discourse and learning interaction to encourage transformative communications and knowledge construction. 
Transmedia navigation within the classroom — allowing student to choose from the wide range of electronic 
and traditional media for interaction with curricular content — is recommended to support engaging 
instruction that provides students with opportunities to think, problem-solve, and engage in critical discourse. 
The authors contend that a design-based research paradigm can guide in extension of theory and practice that 
will bring change to schools in the form of student-driven classroom technologies and enhanced learning for 
students who are motivated to by access to ICT for learning interactions, as well as those who prefer to 
interact with content using more traditional classroom media. 
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ABSTRACT 

Studies indicated Web 2.0 technologies can support learning. Then, integration of innovation may create concerns among 
teachers because of the innovative features. In this study, the innovation refers to Web 2.0 technology integration into 
instruction. To help pre-service teachers make the best use of the innovation in their future instruction, it is essential to 
examine the patterns of their concerns about integrating Web 2.0 technologies into their practice. About 350 future 
teachers participated in the study during the spring semester of 2012 in a northern university of Taiwan. The Stages of 
Concerns (SOC) Questionnaire was adopted to assess seven stages of concern: awareness, informational, personal, 
management, consequence, collaboration and refocusing. The findings of this study indicated the pre-service teachers’ 
concerns as a whole were most intense respectively in the stages of informational, personal, and collaboration. A few 
implications were indicated for teacher education programs to alleviate the concerns. 


KEYWORDS 
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1. INTRODUCTION 

The advantages of high availability Web 2.0 technologies have been advocated for use in education during 
the latest few years. Studies indicated Web 2.0 technologies can facilitate learning by connecting school 
education with informal learning (Ravenscroft, et al., 2013). However, integration of innovation is destined 
to generate concerns among teachers because of the innovative characteristics (Fuller, 1969). When an 
instructional innovation takes place, teachers usually have concerns that may either facilitate or hinder the 
future implementation of the innovation (Hall & Hord, 1987). Fuller (1969) first explained concerns in an 
organized approach and defined concerns as people’s feelings towards an innovation. Concerns are 
developmental; the categories of concerns are centered on self, task, and impact on students (if the context is 
in education settings.). Internal concerns (early concerns) contain concerns related to self; external concerns 
(later concerns) refer to concerns related to task implementation and the impact on student learning (Fuller, 
1969). 

In this study, the innovation refers to Web 2.0 technology integration into instruction. The study specified 
Web 2.0 technology as the technologies which can allow users to co-create some work synchronously or 
asynchronously together, for example, the tools of wikis (i.e., pbworks), concept mapping (i.e., bubbl), 
presenting (i.e., prezi), and so on. To help future teachers make best use of the innovation in their future 
teaching task, it is essential to examine the patterns of their concerns regarding technology integration. With 
the knowledge of their concerns, teacher education programs would be able to develop appropriate 
supporting resources to help the would-be teachers ease and disburden their concerns. Therefore, this 
investigation studied the trend of pre-service teachers’ concerns. It is hoped that the results will help 
highlight the importance of understanding and addressing the concerns in promoting Web 2.0 technology 
integration in education. 
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2. THE STUDY 

The purpose of this study was to investigate pre-service teachers’ concerns on Web 2.0 technology 
integration. The research question was “What are the patterns of the pre-service teachers’ concern regarding 
integration Web 2.0 technology into instruction?”. 

2.1 Methodology 

2.1.1 Participants 

Data were gathered from 350 pre-service teachers who were medium-level users of Web 2.0 technology and 
who attended a teacher education university in north Taiwan during the spring semester of 2012. The 
population consisting of 38.5% of male students and 61.5% of female students, is fairly representative of the 
population of pre-service teachers. The pre-service teachers were invited for research participation at their 
class meetings. Those who agreed to participate were given the consent form and one survey, and they spent 
about 15 minutes filling out the questionnaire. 

2.1.2 Instrument 

This study adopted the Stages of Concerns Questionnaire (SoCQ) (Hall, et al., 1977) that identifies the 
intensity of the seven stages of concern related to a person’s concern regarding an innovation (Hall, George, 
& Rutherford, 1986). Based on Fuller’s theory of three-category concerns. Hall, George, and Rutherford 
(1977) developed the SoCQ to assess concerns about innovation. The SoCQ can measure concerns that 
someone can have when experiencing an innovative practice (Hall, 1979). Furthermore, Hall and Hord 
(1987) developed the Concern-Based Adoption Model (CBAM) as the theoretical framework of the seven 
stages of concerns. The SoCQ has been used in studies of in-service and pre-service teachers during the last 
two decades (i.e., Al-Rawajfih, Fong, & Idros, 2010). Therefore, this study adopted the SoCQ as the 
instrument. 

The seven stages are awareness, informational, personal, management, consequence, collaboration and 
refocusing. Each stage contains five items, and totally there are 35 items listed in a mixed order, using an 8- 
point Likert scale from “not true of me now” (0) to “very true of me” (7). The higher number the stage has, 
the higher concern there is. The validity has been examined in studies (i.e., George et al., 2006). The 
Cronbach alpha coefficients range from .64 to .83 for the seven stages (Hord, et al., 1987). 

2.1.3 Data Analysis 

All data was coded and analyzed with SPSS 20. Descriptive statistics were used to present the data gathered 
from the SoCQ. Raw scores for each sub-scale in the SoCQ were tallied and converted to normed percentiles. 

2.2 Findings 

Table 1 listed the demographic statistics of the percentiles for the seven stages for the pre-service teachers. 
Figure 1 illustrated a line chart of the averaged percentiles for the pre-service teachers and the concern 
profile. According to Table 1 and Figure 1, the mean percentiles for the seven stages were between 37 and 
66. Table 1 indicated the teachers had the most intense concern mean percentile in Stage 1 (information, 
close to 70%), Stage 2 (personal, 59.97%), and Stage 5 (collaboration, 58.43%), which looked like a two- 
humped profile. Regarding the least intense concern, it indicated Stage 0 (awareness) had the least concern 
level (37.06%). Figure 1 illustrated that the general pre-service teachers focused on their concerns in Stage 1 
(information), next Stage 2 (personal) and Stage 5 (collaboration). The levels of concerns dropped in Stage 3 
(management ) and Stage 4 (consequence), peaked again in Stage 5 (collaboration) and dropped again in 
Stage 6 (refocusing). 
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Table 1. Descriptive statistics for stages of concern for the total participants 


Stage 

MeantSD 

Minimum 

Maximum 

0 (awareness) 

37.06±12.57 

11.43 

82.86 

1 (informational) 

66.66±16.17 

20.00 

100.00 

2 (personal) 

59.97±15.86 

0.00 

100.00 

3 (management) 

50.11±16.91 

0.00 

100.00 

4 (consequence) 

53.34±13.60 

0.00 

88.57 

5 (collaboration) 

58.43±17.43 

0.00 

100.00 

6 (refocusing) 

46.86±14.31 

0.00 

88.57 


n=346 



Eigure 1. Stages of concern for the total participants 


The findings of the study brought to light some findings on the status of pre-service teachers’ concerns on 
Web 2.0 technology integration. First, the concerns peaked at Stage 1 and 2, and afterwards they dropped 
gradually until Stage 5 when the concern level peaked again. It implies most of the pre-service teachers had 
some general awareness of the innovation, and they had interest in learning the general characteristics of 
Web 2.0 integration. According to George, Hall, and Stiegelbauer (2006), at the informational stage, teachers 
usually are not worried about themselves regarding implementing the innovation. They are more interested in 
the general aspects of the innovation in a selfless way. The result may imply most of the pre-service teachers 
had little knowledge of Web 2.0. As such, teacher education programs should provide pre-service teachers 
with the Web 2.0 information, resources, workshops or even courses to help pre-service teachers know the 
general features of Web 2.0, the effects of the technology on learning, and time, skill and equipment 
requirement of the technology integration. The main goal would be to deal with their concerns at the 
informational stage and help them progress to the next higher-level stages of concern. Besides, concern level 
higher at the personal stage implies that the pre-service teachers’ concerns focus on how they will be 
influenced by the requirements of the Web 2.0 technology integration, and they wonder if they have the 
ability to implement the integration. In a word, their concerns center on the impact of the innovation on 
themselves. Therefore, after providing the pre-service teachers with sufficient information and resources on 
Web 2.0 integration, teacher education programs should develop these future teachers’ self-efficacy by 
requiring teachers educators to appropriately embed the technologies into their regular instructional activities 
and create opportunities for the pre-service teachers to experience and practice. 

Another hump the concern profile displayed was Stage 5 (collaboration). That implies that the pre-service 
teachers have another concern focus which is on how to work with others (including their colleagues, parents 
and their students). To alleviate the concern, teacher education programs can provide cases for the pre- 
service teachers for reference and may require teacher educators to create collaborative projects or adopt 
collaborative activities for the pre-service teachers to experience. 
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Namely, the pre-service teachers had the most intensity of concern on the informational and personal 
stages, which corresponded with other studies (i.e., Liu & Huang, 2005). During the early phases of an 
innovation, teachers’ concerns tended to exhibit high in Stage 0-2. On the other hand, the pre-service teachers 
held high concern levels in Stage 5, which contradicted with Liu and Huang’s (2005) study. That may be 
related to the culture in the research context. The pre-service teachers may have had little or limited 
collaborative learning experiences themselves through the period of their education. Therefore, it makes 
sense that when they were surveyed about collaborating with colleagues to integrate the technology, they 
expressed high levels of concern. Future study will need to clarify. 

The study has a few limitations. The results of the study came from a limited sample size and from pre- 
service teachers in an East Asian country which made the result unable to be generalized to other types of 
contexts. Second, the results indicated different intensity and types of concerns, but did not investigate the 
relationship of the concerns and personal characteristics. The reasons behind the phenomena tended to be 
inferred from survey data. To make assumptions accurately, future study is suggested to adopt qualitative 
research methods to make up the incompleteness. 


3. CONCLUSION 

Although this study has some limitations, it brought to light some findings on the status of pre-service 
teachers’ concerns on Web 2.0 technology integration. The findings of this study provide some direction for 
teacher education programs to effectively provide pre-service teachers with appropriate resources and 
instruction and hopefully to efficiently disburden pre-service teachers’ concerns, before they jump in the 
teaching career. 


REFERENCES 

Al-Rawajfih, K., et al, 2010. Stases of Concern in Inteeratins E-Leamins in Discovery Schools. Asian Social Science, 
Vol. 6, No. 8, pp. 54-63. 

Fuller, F. F., 1969. Concerns of Teachers: A Developmental Conceptualization. American Educational Research Journal, 
Vol. 6, pp. 207-226. 

George, A. A., et al, 2006. Measuring Implementation in Schools: The Stages of Concern Questionnaire. Austin, TX: 
Southwest Educational Development Laboratory. 

Hall, G. E. and Hord, S. M., 1987. Change in Schools: Facilitating the Process. Albany, NY: State University of New 
York Press. 

Hall, G. E. (1979). The concems-based approach to facilitating change. Educational Horizons, 57, 4, 202-08. 

Hall, G. E., George, A. A. & Rutherford, W. L. (1977). Measuring stages of concern about the innovation: a manual for 
use of the SoC questionnaire. Austin, TX: Southwest Educational Development Laboratory. 

Hord, H., et al, 1987. Taking charge of change (Alexabdra, VA, Association for Supervision and Curriculum 
Development). 

Liu, Y. and Huang, C. (2005). Concerns of teachers about technology integration in the USA. European Journal of 
Teacher Education, Vol. 28, No. 1, pp. 35-47. 

Ravenscroft, A., et al, 2013. Designing and Evaluating Social Media for Learning: Shaping Social Networking into 
Social Learning. Journal of Computer Assisted Learning, Vol. 28, pp. 177-182. 


354 



lADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2013) 


AN EXAMINATION OF TEACHERS’ INTEGRATION OF 
WEB 2.0 TECHNOLOGIES IN SECONDARY 
CLASSROOMS: A PHENOMENOLOGICAL STUDY 


Ling Wang 

Nova Southeastern University 


ABSTRACT 

Web 2.0 tools may be able to close the digital gap between teachers and students if teachers can integrate the tools and 
change their pedagogy. The TPACK framework has outlined the elements needed to effect change, and research on Web 
2.0 tools shows its potential as a change agent, but little research has looked at how the two interrelate. Using a rigorous 
phenomenological research methodology, the “lived experiences” of seven teachers successfully adapting pedagogy with 
Web 2.0 tools were examined giving an in-depth qualitative analysis of how and why teachers integrate Web 2.0 to 
change pedagogy. The research validated the use of TPACK as a framework as well as the use of phenomenological 
research methodology in researching about educational technology. 


KEYWORDS 

Web 2.0, TPACK, phenomenological research methods, digital divide 


1. OBJECTIVES OR PURPOSE 

The main goal of the research was to examine what process teachers use to change their pedagogy to deliver 
effective instruction using Web 2.0 tools. In particular, what decisions were made to adapt the lessons and 
activities, and the reasons behind those decisions were examined. The qualitative tradition of phenomenology 
was used to gather data using the teachers’ voices as they adapted pedagogies with Web 2.0 tools in the 
secondary classroom. The purpose of this research was to understand how the TPACK framework supported 
teachers as they learned the new Web 2.0 tools and adapted them for use. 


2. PERSPECTIVE OR THEORETICAL ERAMEWORK 

Adapting curriculum and pedagogy to incorporate the tools utilized by students on a regular basis has been a 
challenge for teachers (Harris, Mishra, & Koehler, 2010; Kumar & Vigil, 2011; Speak Up, 2011; Wang, 
Ertmer, & Newby, 2004). Web 2.0 tools, such as blogs, wikis, social networking, and bookmarking tools, 
with their ease of use and user friendly interface, may be just the tools that will enable teachers to adapt 
pedagogy (Spivy, Young, & Cottle, 2008). Extensive existing research has focused on how teachers learn 
the technology, but not on how and why they adapt their pedagogy for its effective use in classrooms (Brown 
& Crawford, 2005; Levin & Wadmany, 2008; Linckels, Kreis, Reuter, Dording, Weber, & Meinel, 2009; 
Scrimshaw, 2004; Unal & Unal, 2010). 

Koehler, Mishra and Yahya (2006) have attempted to measure and effect change in pedagogy by 
developing the Technological, Pedagogical and Content Knowledge (TPACK) framework (Eigure 1). The 
framework has been used by researchers (Archambeault, Wetzel, Eoulger, & Williams, 2010; Bull & Eerster, 
2008; Harris & Hofer, 2011; Ward, Lampner, & Savery, 2009; Williams, Eoulger, & Wetzel, 2010) as a 
guideline for exploring teacher professional learning. The framework focuses researchers on the connection 
between teachers’ technological knowledge, knowledge of teaching and how learning occurs, and the needs 
for their individual content areas. 
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Figure 1. TPACK Framework Illustration from http://tpack.org 
The TPACK framework was utilized in this study to frame the research questions: 

1. What Web 2.0 tools were used in the classroom by teachers and students? How were these tools 
being utilized? Why were they being used? 

2. What are teachers’ opinions regarding the technological factors, such as access to web sites, 
computers, or speed of the internet, which either supported or hampered their use of Web 2.0 tools? 

3. How did a teachers’ knowledge of their content impact the decisions they made for choosing 
specific technological tools to teach that content? 

4. How was pedagogy adapted for using the Web 2.0 tools, particularly studying teachers’ opinions on 
which parts of their classroom practices were successful or unsuccessful and why? 

5. What activities with the Web 2.0 tools did teachers feel were easily adapted to teach their content? 
How did they learn about those activities? 

6. How and why did the use of the Web 2.0 tools improve teaching or learning? 

3. METHODS, TECHNIQUES, OR MODES OF INQUIRY 

Because these questions examined teachers’ lived experiences, they lent themselves to a phenomenological 
study. Data was elicited from seven teachers who adapted their pedagogy to use the tools in their classroom. 
The participants participated in two interviews that were taped and transcribed. As per Cilesiz (2011), the 
first interview was an open-ended life history interview, followed by the second interview focusing on in- 
depth reflections based on the TPACK framework of how the teachers adapted their pedagogy to use the 
Web 2.0 tools. The synthesis of their common experiences was then sent to the participants for reviewing and 
input to improve internal validity. The teachers acted as co-researchers, as described by Moustakas (1994), 
clarifying the interpretations of the researcher from the previous interviews. 

4. DATA SOURCES, EVIDENCE, OBJECTS OR MATERIALS 

Phenomenological research requires selection of participants who have significant experience of the 
phenomenon, and criterion sampling of participants who fulfill certain criteria is the most suitable 
methodology (Cilesiz, 2011; Creswell, 2007) for participant selection. Therefore seven participants were 
chosen that utilized Web 2.0 technology in the classroom for at least one semester, taught at least one year in 
the same content area (CK) and grade level (PK), and were willing to share and articulate their experiences 
using the tools in the classroom. 

Participants consisted of one middle school language arts teacher, two middle school computer science 
teachers, one high school and one middle school science teacher, one middle school music teacher and one 
middle school social studies teacher. Participants were given pseudonyms to protect privacy. The first two 
interviews were conducted with each participant with approximately one week in between the two 
interviews. 

Data analysis included horizonalization of the data, or coding the transcripts of the 14 interviews resulting 
in 1 16 codes or meaning units. The meaning units for each participant’s two interviews were merged and the 
textural and structural descriptions describing their experiences were written using imaginative variation. 
Once the data was analyzed, a composite textural description, and a composite structural description, was 
written then combined into a textural-structural synthesis of the experience which was sent to the participants 
for their feedback, and adjusted to reflect their input. 
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5. RESULTS AND/OR POINT OF VIEW 

The textural and structural descriptions of each participant gave a rich description of the teachers’ 
experiences. The essence of the experience of the phenomenon centered around interrelated themes supported 
by the TPACK components as indicated in the textural-structural synthesis of the study. A summary of that 
synthesis follows. 

Technology skills (TK) were obtained by the participants either in formal training classes, or “hit or miss” 
on the job training. Extensive additional personal time was spent by the participants researching, exploring 
or “playing” with and learning both basic technology and the Web 2.0 tools. The additional time spent 
learning or exploring helped the participants overcome some of the technological challenges that came with 
adapting something new. Technical challenges included lack of access to computers, not enough outlets, sites 
locked by firewalls, slowness of the internet at certain times of the day, freezing computers, or software or 
online tools that would not work for specific tasks. A challenge specific to Web 2.0 tools involved keeping 
track of “all the tools, trying to keep passwords the same, keeping a record of all my passwords, and which 
computers I registered which site on (Renee)”. Participants overcame these challenges either through the 
confidence gained from previous training, consulting help files or their colleagues such as the integration 
specialist in the school. 

All the participants felt they had strong background knowledge in their content (CK) with most having 
completed Masters’ programs. They all acknowledged that they were always learning by keeping up in their 
field. This content knowledge enabled them to “... know where things might fit when I teach that section” 
(Debby), and to understand which concepts were best taught with which tools (TCK). Sometimes tools were 
chosen because they made students use their background content knowledge. In addition, the tools gave the 
teachers a diverse set of tools to make teaching the subject more effective (PCK). Sometimes decisions on the 
tools were based on the ease of use of the tools for what the students needed to do with the content (TCK). 
Some decisions were influenced by the fact that technology itself is changing their curriculum. Some choices 
were based on how the tools could be used to highlight parts of the content. Examples are the use of Prezi in 
the science classroom to get the overall picture along with the elements, or in the social studies classroom the 
use of Edmodo to explore a picture, connected to an historical time period, in detail. 

Pedagogical knowledge (PK) varied among the participants. One participant was a second year teacher 
with her pedagogy still in the process of being developed, however, she also had support from the mentor 
teacher, her department chair, and the integration specialist that aided her in developing and fine tuning the 
pedagogy. Another participant was a researcher before a teacher, so was not as aware of pedagogy and what 
it was. However, her content knowledge was so strong that she instinctively seemed to do what her students 
needed to grasp her content without being able to identify it as pedagogy. The remainder of the participants 
had been teaching a number of years, and had fine-tuned their pedagogies before interacting with the Web 
2.0 tools. Strong pedagogical knowledge is knowing how teaching and learning can change, and if that 
change is beneficial (PK). An example of that is creating an Animoto character to “...bring characters to life 
is a different way to express ideas in literature (Debby)” which makes students think in a deeper way about 
what they’ve learned (PCK). Another teacher improved teaching by “ ...being able to present concepts to 
students by showing a lot of pictures based on nature, and then ask them what the songs will be about. They 
see it without you saying it. I use it so that I can talk less and the students are able to form ideas about the 
music before they listen to it (Janece).” Another evidence of teaching and learning changing, as expressed by 
several teachers, is the concept of hearing from all students in the class. As one put it, “... all the kids had to 
respond. ..so I think it was better than being in the classroom where you only call on a few students (Debby).” 
As one teacher summarized, with Web 2.0 tools, pedagogy can change “...from teacher centered to a 
facilitator as a guide on the side, to lead students in the right direction. They are more engaged in finding 
their own information, and when sometimes they find the wrong information, you have to step back and 
guide them to other websites or to what other kids are blogging, and guiding them to rethinking their thinking 
(Teresa).” (TPK) 


6. SCIENTIFIC OR SCHOLARLY SIGNIFICANCE OF THE STUDY 

Previous research has focused on teacher professional development in instructional technology with 
pre-service teachers (Brown & Crawford, 2005; Levin & Wadmany, 2008; Linckels, et al, 2009; Scrimshaw, 
2004), or with in-service teachers on individual tools such as Webquests (Unal & Unal, 2010). 
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This research extended previous studies to include an in-depth view of in-service teachers’ experience 
with a variety of Web 2.0 tools (Pan & Franklin, 2010) and focused on TPACK (Kohler, et al, 2006). This 
research points to ways to make the change process easier for teachers, administrators, and students. 

In addition, utilizing a rigorous phenomenological methodology supported Cilesiz’ (2011) conceptual and 
theoretical framework of the methodology as a primary research method for educational technology. The use 
of the TPACK framework during the interview process further supported that framework as an additional 
tool for research with in-service teachers. 
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ABSTRACT 

This study explored students and instructors’ perceptions and experience of technology affordances in an technology- 
enhanced Active Learning Classroom (ALC) to promote students’ collaborative problem solving. Multiple case studies 
were conducted. Eive classes of 92 students and five professors participated in this study. The data sources were class 
observations, interviews, and pre- and post-surveys. The study showed that students’ self-efficacy and confidence in 
completing problem-solving tasks increased over time. Additionally, it was found that some professors used the ALC to 
its potentials while others used it minimally. While both students and professors agreed about numerous benefits of ALCs 
for learning and instmction, how technology was used depended on the perceived purpose, needs and meaningfulness of 
the instmctors. 


KEYWORDS 

Active Learning Classroom, Technology, Affordances 


1. INTRODUCTION 

In recent years, there has been a growing interest in designing technology-enhanced Active Learning 
Classrooms (ALCs) to facilitate collaborative learning (Kim & Hannafin, 2010, Montgomery, 2008). 
Research shows that “space matters” to enhance students’ learning (Montgomery, 2008; Oblinger, 2006). 
ALCs are equipped with technology, such as large round tables, microphones, boards around the classroom 
edges, and large LCD screens, to promote interactive, student-centered learning (Walker, Brooks, & Baepler, 
201 1). In an ALC environment, students are placed on the spotlight of learning, engaging in critical thinking 
and problem solving through furniture arrangement and setting design while a professor is the coach instead 
of an information transmitter. 

Evidence suggests that classroom features influence how students learn and how instructors teach (Brown 
& Long, 2006; Chism, 2006; Chism & Bickford, 2002; Lomas & Oblinger, 2006; Oblinger, 2006). Studies 
show that the ALCs have a positive impact on student learning outcomes (Brooks, 2010; Walker et al, 2011; 
Whiteside, Brooks & Walker, 2010), enhance students’ conceptual understanding, improve their problem- 
solving skills and attitudes, increase motivation (Beichner et al, 2007; Dori et al, 2003), and enable 
instructors to align their teaching methods with classroom features accordingly (Walker et al, 2011; 
Whiteside et al, 2010). While research suggests that space matters, it also indicates that instructional 
approach matters. Some studies indicated that students made more gains in a team-based learning 
environment than in a lecture-based environment in the same ALC setting (Walker et al., 2011). However, 
the previous studies did not investigate what factors in an ALC influence or motivate student learning. Most 
of the research we found focuses on studenfs’ achievemenfs (e.g., course grades, quizzes, exams, and 
homework) and less on sfudenfs and professors’ perceptions and experience of ALCs. 
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2. PURPOSE 

The purpose of this study was to investigate the impact of the ALCs on students’ learning and instruction, 
their perceptions of technology affordances, and their experience of technology for facilitating collaborative 
learning. It was also to understand instructors’ instructional behaviors and instructional decisions in using 
technology. Ecological psychology (e.g., Gibson, 1979; Gibson & Pick, 2003) argues that individuals are 
information detectors who are capable of perceiving affordances in the environment and how they become 
apprised of these possibilities for action (Young, Barab Garrett, 2000). Central to ecological psychology 
are the concepts of affordances and effectivities (Gibson, 1979; Gibson & Pick, 2003). The literature suggests 
that the space does not only facilitate students to collaborate on projects and engage in inquiry learning but 
also allows instructors to modify their teaching methods accordingly. It appears necessary to explore the 
following questions: 1) How do instructors choose to use technology to carry out their instruction? 2) What 
technology do students use to facilitate their collaborative problem solving? 3) What is the impact of ALCs 
on students’ motivation and self-efficacy in solving problems? And 4) What are students’ and instructors’ 
perceptions regarding ALC affordances? 


3. METHOD 

Multiple case studies were conducted for this reseaerch. Five classes of 92 students (both undergraduates or 
graduates) and five professors from various disciplines (meteorology, biology, zoology, political science, and 
chemistry) participated in the study. Four professors were also interviewed. Each participating class was 
regarded as an individual case unit (Stake, 2005; Yin, 2002) and was examined through class observations, 
interviews, and surveys. Observations were conducted at different points of the semester, focusing on 
instructors’ teaching approaches, class activities, use of technology in ALC, and the interactions between 
instructor and students and among students. The surveys were administered at the beginning and the end of 
the semester, asking students’ perceptions about: intrinsic motivation, problem-solving confidence, and 
problem-solving skills related to their subject domain. Descriptive statistics were conducted on the 
observation data, which were also qualitatively analyzed and coded. The interview data were coded, 
interpreted, categorized, and triangulated with the observation data to identify themes. 


4. EINDINGS 

Figure 1 and Figure 2 demonstrate two contrasting cases - Case 1 (Zoology: 14 students, undergraduate and 
graduate) and Case 5 (Organic Chemistry, 24 undergraduate) with the percentage of time distribution in 
group work, individual work, lectures, student-instructor interactions, student presentations, and quizzes. 
The two representative cases were in contrast regarding the time spent for the class activities. Case 1 was 
much more balanced with time spent between instructor lecturing and students’ activities, while Case 5 was 
largely dominated by expository and lectures. The other classes fall under either more student-centered 
learning (e.g.. Case 1) or less student-centered learning (e.g.. Case 5). At the same time, it was observed that 
Case 1 took full advantage of the ALC technology to promote their collaborative problem solving, while 
Case 5 only limited technology use to big screens, round tables and iPads. Furthermore, Case 1 instructor 
scaffolded students’ problem-solving through many case studies and discussions, while Case 5 instructor 
mainly used technology to demonstrate concepts which were difficult for students to understand and to 
illustrate complex relationships among the concepts. 

The survey results indicated that students achieved significantly higher confidence scores in the posttest 
(Mpre=308.10, SD=44.34; Mpos,=332.86, SD=32.58; p< .05), but there were no significant differences in the 
measures of perceived intrinsic motivation problem-solving skills. 

Most students interviewed agreed that “the layout of the room and the synthesis of technology make 
group learning possible” (e.g., Case 4) and prompted them to work together more freely and to interact with 
the instructor more at ease. Some students (e.g., Case 5) indieated that they had “learned from multiple views 
and how to reaeh consensus” through collaborative learning. 
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Some other students (e.g., Case 4) mentioned that the app that downloaded to the iPad helped them to 
learn difficult concepts through the display of molecule structures and dynamic interactions of molecules. 
Overall, the students summarized the value of the ALC as “interaetive, engaging, and effeetive.” 




Eigure 1. Class Time Distribution of Case 1 Figure 2. Class Time Distribution of Case 5 

However, there were some concerns regarding the use of the ALC. One of the concerns was that there 
was often short of time for group activities given the current class structure (meeting 1 hour per class, 3 times 
per week), leaving little time for intensive group discussions or projects. Some other students also expressed 
the concern regarding those group members who did not prepare before class or contribute minimally to 
group projects. The interviews also indicated that some professors did not use technology to its full 
potentials, or some professors did not have adequate training in using the technology. 

The instructors commonly recognized that by teaching in an ALC they were no longer confined to the 
podium; and that they could walk around easily to monitor and guide group activities, answer questions any 
time, and provide timely feedback. It gave them ample opportunities to identify students’ difficulties during 
class. In addition, the three instructors (Case 1, Case 2, and Case 4) really enjoyed the technology, which 
helped them to illustrate abstract concepts more easily. One instructor said that the document camera was 
very helpful for him to illustrate the domain content related to atmospheric physics, while the instructor of 
biochemistry indicated that the app downloaded to iPads helped him to illustrate the structure and the 
interactions of chemistry molecules to his students and to engage his students in manipulating and observing 
the molecules in 3D view, which was consistent with the feedback obtained from the students. The ALC 
technology helped the instructors to consider developing instructional strategies or learning activities that 
were aligned with the characteristics of ALCs (e.g., group activities or case studies). A professor said that the 
ALC was not just about technology or space, but rather “a new way of thinking of learning and instruction.” 
However, there were concerns about not having enough time to cover the content. A professor believed that it 
was his responsibility to “build the mental structure for the students; and then they could fill in the gaps.” 
Therefore, he found himself in a dilemma of offering more time for students’ group work or spending more 
time covering the course content. Two instructors indicated that appropriate use of the ALC depended on 
course content and class size. They believed that ALC seemed more suitable for small classes where group 
activities were more possible due to the constraints of time and class size. Some professors indicated that 
they did not know how to use all of the technology. As they learned more about different functions of the 
technology, they might be able to come up with more instructional strategies and student activities. 


5. DISCUSSION AND IMPLICATIONS 

It was found that there was a wide gap between professors who used the ALC technology to its full potential 
and those who only used it minimally. Some professors used the ALC more frequently for group work and 
collaborative activities, while others predominantly used the room for lectures, worrying about lack of time 
to cover course content if time would be spent for group activities. According to ecological psychology, ALC 
provides affordances for improving learning and instruction, but it relies on the ability of users to take 
actions. The users must perceive the meaningfulness of technology and consider their needs (Gibson, 1979; 
Gibson & Pick, 2003). In this research, the professors perceived the use of technology in different ways 
according to the nature and content domain of a course. 
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They were the ones who determined how to drive a class (Gibson, 1979; Gibson & Pick, 2003), which 
was contingent upon numerous factors, such as content domain, class size, course structure, and their 
personal views of learning and instruction. 

The findings inform us that in the ALCs, active learning does not happen automatically; effective 
instructional design strategies are needed to make active learning happen. The use of ALCs requires a 
fundamental paradigm shift on both professors and students: a new way of viewing knowledge, learning and 
instruction. The study implies the necessity to provide extensive examples and trainings for professors on two 
dimensions (technological and pedagogical) and to reconceptualize “time” and restructure class schedule 
(e.g., changing three meetings per week into 1 or 2 meetings to provide students more time for group work) 
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ABSTRACT 

Game-Based Learning (GBL) is a promising and engaging tool for STEM (Science, Technology, Engineering, and 
Mathematics) learning. How GBL promotes content learning and mastery is unknown, however. For GBL to be more 
than an engaging tool for delivery of basic knowledge, it must be designed to achieve the goals of Project-Based Learning 
(PBL). PBL achieves content mastery by using principles of inquiry to promote authentic learning. The challenge is to 
keep GBL engaging while incorporating inquiry strategies into gameplay. This can be achieved through immersive 
micromanagement and virtual world games that incorporate content mastery objectives into player strategies for 
advancement in the game's plot. Complexity introduced through evolving game scenarios can push players towards 
decisions guided by content lessons. Different applications of GBL in classroom and informal science settings present 
unique challenges and opportunities. Team play and competition can enhance PBL elements and increase cognitive 
outcomes in the classroom, while flexibility and portability across multiple formats and delivery mechanisms is crucial to 
success in informal science education (ISE). This paper provides guidelines for the development of GBL for both formal 
classroom settings and ISE. 
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Constructivism; Game-Based Learning; Project-Based Learning; Problem-Based Learning; Serious Games; Informal 
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1. INTRODUCTION 

Game-based learning (GBL) is proving to be an increasingly promising and engaging tool for STEM 
learning. The success of Fold It (Cooper et al., 2010), Phylo (Kawrykow et al., 2012), and Eterna (Eterna, 
2011) are showing that players will independently engage with games that achieve real-world results. Games 
such as New Century Energy and many others are showing that GBL can achieve authentic learning in the 
classroom (Hoge et al., 2012; Hoge & Hughes, 2011; Ahn & Dabbish, 2008). Authentic learning is learning 
at high levels of bloom’s taxonomy by using the same tools and methods as experts use in the real world. 
This type of learning is fundamental to the pedagogical philosophy of constructivism and has been shown to 
be the best way to teach STEM content and skills (Slough et al., 2004). Constructivism also provides the 
basis for achieving learning outcomes through GBL (von Staalduinen & de Freitas, 2011). 

Authentic learning is about inquiry and problem solving. For GBL to be more than an engaging tool for 
delivery of basic understanding and application, it must be designed to achieve the goals of inquiry learning 
as demonstrated by project-based learning (PBL) models. The challenge is to keep GBL engaging while 
incorporating inquiry strategies into game play. This can be achieved through immersive micromanagement 
and virtual world games that incorporate content mastery objectives into player strategies for advancement in 
the game's plot. Complexity introduced through evolving game scenarios can push players towards decisions 
based on content knowledge acquisition, analysis, and creation of real outcomes. Team play and competition 
can also enhance the PBL elements and increase cognitive outcomes in the formal classroom but are difficult 
to accomplish in informal science settings. A virtual world approach across multiple formats and delivery 
mechanisms can overcome these limitations in informal settings. 
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2. GAME-BASED LEARNING AS PROJECT-BASED LEARNING 

Games are successful PBL because they tend to put the learner in the role of decision-maker, pushing players 
through ever harder challenges. Learning is authentic because it is accomplished through trial and error (gee, 
2003). Games also provide immediate feedback on the player’s strategy which encourages exploration and 
experimentation (Kirriemuir, 2002). Games are effective for multiple learning styles and levels of 
understanding since they can challenge players to their specific prior knowledge and skills (gentile and 
gentile, 2005). Another aspect of gaming that is particularly important for stem education is that games allow 
players to fail, but reward persistence and risk (Gerber, 2012). Most stem education focuses on what is 
already known. This does not prepare students for the real world where the focus is on elucidating the 
unknown, and where new problems arise constantly. Even most PBL presents projects or problems for 
students with known answers and proscribed strategies for accomplishing the task. While these hands-on 
approaches do provide a more constructivistic learning experience, they are not giving students the most 
important skills for stem applications: risk taking, persistence in the face of failure, and creative problem 
solving. In his 2010 talk to ted.com, Thom Chatfield outlined seven key points about games that can be 
leveraged to adapt them to learning environments (Chatfield, 2010). Games provide immediate feedback. 
They challenge players with a sense of uncertainty, but they reward effort, both small and large. Games 
should incorporate short and long-term goals, and use avatars to immerse players in an “experience 
system Research into neurology and learning can guide the development of games that can create 
“windows of enhanced attention ”. And, games can involve players with social contexts in a variety of ways 
such as collaboration, competition, and identification with shared goals and values. 

Numerous challenges must still be overcome before GBL can reach its full potential, however. Though 
numerous models exist for achieving learning outcomes through GBL (van Staalduinen &. de Freitas), it is 
not fully understood exactly how GBL accomplishes authentic learning. Formal and informal science 
environments present unique challenges and opportunities for GBL. In both settings, for GBL to become 
successful PBL, games should: 1) simulate real-world scenarios; 2) combine aspects of strategy games and 
construction and management games; 3) allow players to control the progression of the game through their 
game play and decisions; 4) be played by teams of players formed as cooperative learning groups; and 5) be 
competitive across multiple levels of difficulty. Variations in difficulty should be incorporated into lessons 
that students must master in order to open options within the game. Players should be required to answer 
questions from each lesson, with missed questions affecting the efficiency of outcomes/success. Questions 
should be based on NAEP guidelines for basic, proficient, and expert levels. Lessons and questions should be 
designed to lead players to game play decisions that require understanding of STEM concepts. The 
complexity of the game should push students to seek out content knowledge in order to advance (this makes 
the experience constructivist project-based learning). A meta-site should be made available outside of game 
play for students to supplement their learning for subsequent game play. Lessons within the game must be 
carefully crafted to encourage learning through inquiry. 

Over the years, PBL has taken on different meanings depending on whether the project involved is long- 
term with multiple learning objectives, or short term, centered on a specific problem. GBL should provide 
aspects of both project-based and problem-based learning. Both types of PBL emphasize “constructing 
understanding of important science concepts as they inquire into a real life problem” (Schneider et. al, 2002). 
PjBL requires students to solve problems through hands-on engagement in the entire process of scientific 
inquiry, from researching the problem, forming hypotheses, designing tests, performing experiments, and 
communicating results. PbBL can more effectively focus on specific content objectives and skills. Players 
should be required to solve problems on many levels to advance. The flow of gameplay must be designed so 
that experienced game players and novices with interactive media will encounter problems appropriate to 
their levels. Complexity within the game should encourage players to develop strategies and to adapt their 
strategies throughout the game. A meta-site should be made available to players to conduct research outside 
of game-play. In the Classroom, the formation of cooperative learning teams should be structured to 
encourage each team member to take on specific roles within the game and to take on specific research 
assignments for the team. In informal settings, through virtual worlds, players can benefit by cooperating 
with other players, and though competition. Each enhance the cognitive outcomes of the game. Players can 
compete within the game, or through scores based on multiple factors. The total score should be some sum of 
a knowledge score and a strategy score. 
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These scores should be weighted equally and converted into points (50/50). The knowledge score could 
be derived from the number of questions answered correctly divided by the number of questions available at 
that point of the game, for instance. The strategy score should be derived from the specific success criteria of 
the game. 

NCE is an immersive micromanagement computer game which provides a transformative learning 
experience to 8-10* grade students. NCE combines aspects of strategy games and construction and 
management games, requiring players to build energy companies that must meet the needs of cities 
throughout the US over the next 50 years. Teams of players are formed as cooperative learning groups which 
compete across three levels of difficulty. Variations in difficulty are incorporated into lessons that students 
must master in order to open options within the game. Players must answer questions from each lesson, with 
missed questions affecting the efficiency of plants built. Lessons and questions are designed to lead players to 
game play decisions that require understanding of physics, chemistry, earth science, and math concepts. 
Lessons within the game are carefully crafted to encourage learning through inquiry. NCE was strategically 
designed to encourage students to seek out content knowledge and apply it to real world scenarios. The 
expectation was that rewards for seeking out and applying knowledge would encourage student’s acquisition 
of content knowledge, and provide the perspective and skills necessary for the application of this knowledge 
within and outside of the game. Comparisons of score components to total score indicate these expectations 
were met (Hoge et al., 2012). 

A new game similar to NCE is now being developed for informal science settings. As with NCE, 
development of a smart game for informal science settings requires a complex collaborative process. This 
provides opportunities to develop innovative game design and cognitive strategies, but also generates 
unpredictable challenges and opportunities. GBL for ISE has the potential to add many new aspects, 
including multiple platforms for the game, interactive 3-D graphics, mobile apps for player research, and 
online extensions. The game will take place in a virtual world allowing players to advance at their own pace 
as they develop and pursue gameplay strategies. This experience system of advancement keeps the action 
consistent throughout the game. This also allows for multiple levels of engagement with the game. Casual 
players will not become lost in the depth of a lengthy game while seasoned players will find a challenging 
experience to come back to. To accomplish this, the game will be segmented into short challenging mini- 
games. While a complete play-through of one segment will be relatively short (roughly 20-30 minutes), 
players can shorten this time by facing one challenge at a time, or increase their engagement with the game 
by progressing to other segments. Multiple segments will compartmentalize over-riding themes in the game, 
but will each provide content in all areas of STEM and challenge players on multiple cognitive levels. The 
segments will be equal in length and complexity. Player experiences with each segment will be uniquely 
different each time it is played. This will be accomplished by randomness in world attributes, competitive 
and cooperative competing businesses, and a variety of obstacles for the player to overcome. Each game 
segment can be played multiple times with different outcomes. With a gameplay of 20-30 minutes assumed, a 
single scenario can be played very differently each time. These multiple scenarios purposely steer repeat 
players to progressively higher cognitive levels as they gain knowledge and skill. Players can initiate new 
projects whenever they want. The “development mode” of the game will not be focused on one city or 
region. As in NCE, the player has the option to choose from multiple regions around the world per project, 
depending on what resource they are investing in and what is available to them. The player will be directed 
within the game parameters by showing them options for investment and building. Although players are 
given a list of game options, the positives and negatives of each area is initially hidden. Costs, access to 
resources, and risks to the environment are all unknowns initially. The player can mix risk with reward in this 
system. This makes the game a “real world” model. The player could potentially take a huge risk and hope 
that they are successful, or the player could choose to do research before making an investment of time and 
resources. This operative method will produce better learning because the feedbacks to player choices will be 
immediate and consequential. If the player is not forced to do some research, it doesn’t stress the importance 
of understanding the risks before taking them. Giving a player the option to fail and return better prepared 
will teach them the value of preparedness. 
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3. CONCLUSION 

No component of a PBL strategy can be successful by itself, and this is true for GBL as well. NCE was 
designed to take smart gaming to the next level by encouraging the promises of PBL through GBL. Initial 
analysis of NCL results indicates that the design strategy of NCL can achieve these goals. Analysis of game 
scores show that successful teams were likely to seek out content knowledge, and that they used that 
knowledge effectively to achieve high scores. NCL fulfills the promise of PBL/GBL by not only engaging 
students in content lessons, but also in the “construction” of advanced knowledge and skills. As the game 
becomes more complicated, players must not only seek out information on STLM topics, but also conceptual 
understanding and big-picture perspective. NCL is more than the game, however. The combination of the 
NCL game with teacher training, curricular support, community resources, and the incentives of competition 
make NCL a transformative approach which could shift the focus of GBL towards a tool for inquiry rather 
than simply an engaging content delivery tool. Transitioning into a virtual world scenario for ISL provides 
new challenges and opportunities, but the same pedagogical principles for authentic learning apply. 
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ABSTRACT 

This paper updates the efforts to educate blind students in higher education in the digital age and describes how to 
support the development of mental models in learning through tactile learning and 3D -printing technology. It cites 
research documenting a drop in Braille literacy along with the growth in use of digital technologies by blind students. It 
identifies technologies, strategies, and techniques used to address barriers to progress faced by a student majoring in 
computer and information systems, where learning to program a computer and design databases were requirements for a 
degree. 
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1. INTRODUCTION 

During the digital age literary Braille usage has been dropping and fresh barriers for visually impaired 
learners are posed through online education and new visually-oriented web page interfaces (Thompkins, 
2010; Perry, 2011; Avin, 2010). Novel digital technologies provide access to text in audio format, such as 
screen readers, ebooks, and mp3s (Avin, 2010). We focus here on the needs of the severely visually impaired 
(SVI) student and on the teaching/learning process, according to the Universal Design of Learning (UDL) 
(CAST, 2011) first principle of providing “multiple means of representation.” 

The SVI student must transcend sole reliance on audio inputs to confirm concepts of structure and 
relationships vital in learning about information technology. After a model (hypothesis) of a structural 
relationship is formed, it “can then be tested by the agreement or disagreement of predictions based on it with 
new facts which are discovered and its merits assessed by the degree of modification which it requires in 
order to meet these new facts.” Craik (1943). How should a relationship be modeled for a blind learner? 
Abidin et al (2012, 2013) explored blind persons' mental models of web pages and investigated the 
relationship between users' mental models and usability problems they face when using a screen reader and 
found that blind users possess a functional or structural mental model or a combination thereof. 


2. INSTRUCTIONAL ENVIRONMENTS 

Here are the instructional environments used for teaching the SVI student at Robert Morris University 
(RMU), with their advantages, disadvantages, and special provisions or accommodations; 

2.1 Ordinary Classroom Setting 

The advantages are that (a) the student is integrated with other (sighted) students, as he or she would be in a 
work situation, (b) the student receives all the auditory information that all students in the group receive. 
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The disadvantages of this setting are that any diagram on the board, drawn by the instructor or other 
students, or distributed to all students on a sheet of paper, are useless for the SVI student. Sighted students in 
the classroom may become impatient for attention if attention to the SVI student’s needs requires more than a 
few seconds. Accommodations are to arrange for a sighted student sitting next to the SVI student to interpret 
situations and explain diagrams or work on the board. Printed or posted instructions that are adequate for 
sighted students are often inadequate for an SVI student, especially where the student must learn to use a new 
computing environment. Occasional one-on-one meetings (perhaps 1 per week) can be scheduled to assure 
that the SVI student has adequate opportunity to ask questions and have them satisfactorily answered in 
detail. These occasional meetings offered an opportunity for assessment through dialog to help the instructor 
track the SVI student’s progress and to verify that all instructions on how to use a computing environment are 
usable. Relying solely on class meetings and student questions during class is likely to be inadequate. 

2.2 One-on-one Classroom Setting 

The advantages are that (a) the student receives individual attention and support. The teacher doesn’t have to 
worry about pacing impacts on other students, (b) The instructor has the opportunity to maintain running 
class notes for both to consult subsequently. Sometimes the student needs time to refine a screen navigation 
strategy which can then be documented in the Class Notes, (c) student and instructor discuss strategies as 
needed to solve technical challenges. Disadvantages may be that (a) the instructor may not be patient enough 
for the SVI student and may prompt him/her on the basis of what the instructor can see when the student 
should be allowed to work through the situation at his/her own pace and learn to detect problems. 
Accommodations include class notes, joint instructor/student evaluation of technology options, such as screen 
readers. 

2.3 Over-the-phone Setting 

Where both student and instructor have access to a multiuser server, an over-the-phone instructional setting 
can be practical in exploring the learning of programming. The advantages are, ironically, that the instructor 
cannot see the SVI student’s screen and is not physically present and thus cannot prompt the student based on 
what the instructor can see. The student must be self-sufficient. Disadvantages are that the instructor cannot 
see the SVI student’s screen and is not physically present and thus cannot help resolve certain problems that 
may arise and may even lead to termination of the session. Accommodations include multiuser access to 
systems, and two models of this were employed: (a) multiuser access to an InterSystems Cache ObjectScript 
server and (b) access to two Linux virtual machines in a virtual machine network. 


3. SUPPORT TECHNOLOGIES AND PROCEDURES 

This section describes the learning support strategies adopted in teaching INFS3141 (Advanced M 
Programming) and INFS4953 (Intrusion Detection). When the student chose to major in Information 
Systems, he faced a requirement for completing two courses in programming in the same language. 

Eventually he decided on M programming from the options available. What were the advantages and 
disadvantages of using the InterSystems Cache ObjectScript application programming interface (API)? Prior 
to working with InterSystems Cache, our SVI student had been counseled to try COBOL. He found this not 
practical. Cache (M Programming) was then recommended as the programming can be carried out with 
single-letter commands and single-letter or two-letter function identifiers and the programs are not as verbose 
as those coded in other computer programming languages. While database capabilities are integrated with the 
programming language, the database does not have to be declared, as is the case with SQL. M databases are 
considered to be NOSQL databases and can support the largest datasets. The industries in which M 
programming is used include, in the U.S., health care and the financial industry (banks and credit unions, 
stock market, and even space research (InterSystems, 2012)). One disadvantage is that the Cache API Studio 
has an editing error indicator which, so far, is not accessible to SVI learners. 
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3.1 Screen Reader Selection and Utilization 

Screen-reading technologies, predominantly JAWS (2013), were employed by the student in this project. His 
experience with Braille had been unsatisfactory so he had meager Braille capabilities. JAWS can distinguish 
between upper and lower case by voice pitch. Because of the significance of case sensitivity in programming 
environments, this is an important capability of JAWS. Experimentation showed that no single screen reader 
met his needs in learning programming. JAWS sometimes does not render all program output on the screen, 
dropping whole sections of software output. This led to the student using a second reader, NVDA (2013), 
which added to the complexity of managing these technologies and comprehending and verifying program 
output, with two screen readers sometimes generating spoken text at the same time. 

3.2 Tactile Access to Diagrams 

A major challenge in educating an SVI student is the prevalence of visual materials in curriculum. Diagrams 
and illustrations are common, delivered ordinarily to students through software, instructional management 
systems like Blackboard, Power Point presentations, through faculty and student drawings on classroom 
chalkboards and whiteboards, graphic material in textbooks and Internet links, and as an object for 
discussions. Eventually it was apparent that the most significant barriers learning were posed by certain key 
conceptual diagrams employed on a variety of courses. Examples are: 

Table 1 . Instmctional Diagram and Graphic Image Examples 
Entity-relationship (ER) diagrams in database design 

Database Relationship Diagram as displayed by Microsoft Access (See Figure 1) 

Unified Modeling Language (UML); tactile access documented by Brookshire (2006). 

Relational database design: table relationships and cardinalities (SQL standard is ISO/IEC 9075-1:2011; see 
Figure 1) 

Models of Cache ObjectScript global (or local) variables as databases (standard: ISO 1 1756; see Figure 2) 
Comparisons of the relational and hierarchical database models in courses in which the student enrolled 

(INFS3141 andINFS3450) 

Maps; accessibility strategies documented by Koch (2012). 

Following UDL principle I led to consideration of audio and tactile feedback to deliver multiple 
representations. Audio feedback for diagrams or graphs was covered by Cohen et al (2006), Sanchez (2008), 
Kennel (1996), Sanchez and Flores (2010), Abidin et al. (2012, 2013). Tactile feedback was covered by 
Petrie et al. (1995), Brookshire (2006), Levesque (2008), Koch (2012). Textures in tactile models was 
covered by Abidin et al. (2012). Haptics was covered by Levesque. The decision was made to pursue static 
tactile models for diagrams. 



Figure 1 . Relational Database Model Figure 2. M Array (Global) Database Model 

The design of 3D tactile models had to consider whether Braille numbers should be displayed on the 
model (Figure 1). The student did not commonly use Braille but could read Braille numbers. Number 
symbols are required before each numeral in an environment where both numeric and nonnumeric symbols 
are used. 

Commonly used Braille did not have a symbol for infinity, so we either had to use a different Braille 
standard or use the letter “m” for many in expressing database diagram cardinality. Brookshire (2006) solved 
this problem simply for his tactile models. For the second model (Figure 2), Braille was not used. 
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Both student and instructor assessments and concerns were required to identify the models or diagrams of 
greatest need and thus prioritize the process of producing 3D models. 

Input from the student on 3D images began with the relational database model shown in Figure 1. The 
student characterized use of the 3D model of a diagram as a novel, unprecedented experience and reported 
that it helped him gain an “accurate interpretation” and that he revised his mental model of database 
relationships based on his experience with the 3D model. The Braille numbers on the original model was 
discovered to be superfluous because (1) the student could interpret the Arabic number representations as 
effectively as those in Braille and (2) mixing numeric and alphabetic symbols required more symbols, and (3) 
ordinary Braille was not designed to present scientific and technical symbols only available in specialized 
versions of Braille, such as Nemeth Braille (AAWB, 1987). 

Learner input regarding the M array model (Figure 2) revealed that he gained new insights from using the 
model. The student practiced two database array traversal functions of Cache ($data() and $order()), while 
consulting the model through touch. $Query() function array traversal was also aided through use of the 3-D 
model. 

The models presented in Figures 1 and 2 were designed in Solid Works and printed using the Fused 
Deposition Modeling (FDM) process by employing a Dimension Elite Printer and ABS (acrylonitrile 
butadiene styrene) industrial plastic. No chemicals such as sodium hydroxide were utilized in removing the 
support materials, but mechanical loading was used to prevent contamination of the models. 

Front edges of raised blocks as well as the base plate for both designs were chamfered and the lower left 
corner of the design illustrated in Figure 1 had a number sign in Literary Braille to establish orientation for 
the SVI user. Design illustrated in Figure 1 had also Nemeth Braille Symbols representing infinity with a 
comma and an equal sign while rest of the raised information was given in either Numbers or Literary 
Braille. The design illustrated in Figure 2 was printed in two pieces and assembled through a snap fit 
connection. 

Even though current EDM models are not cost effective, they can be improved with use of less material, 
also by lowering process times and costs which is more critical than material costs. Reductions in process and 
materials costs due to the new concept of Rep-Rap (Replicating Rapid Prototypers) will make printing 3D 
tactile models economically justifiable. 

Design of inexpensive refreshable (programmable) diagrams may pose another alternative along with 3D 
printed tactile models. These flexible programmable tools can hold a major advantage over their 3D printed 
counterparts with the addition of voice feedback coupled with their tactile content. 

Assessment of the impact of the 3D model of the M array consisted of exercises involving the identical 
array as modeled and of additional array models, in order to discover what learning, if any, had occurred 
through consulting the 3D model. It was apparent that the learner had achieved a generalized model valuable 
for approaching arbitrary alternative array instances. 

3.3 Multiuser Server and Virtual Machine 

RMU utilized Amazon Web Services and built an EC2 server instance for this project. InterSystems Cache 
for Windows (x86-64) 2012.1 (Build 564U) was installed this Windows 2008 R2 server. Cache user accounts 
were created for students and faculty. While Cache can be installed locally for development and testing 
purposes, this server-based multi-user environment allowed faculty and students a real-world experience with 
shared database objects, scripts, and data. 

As an example of software used by an SVI student in the multiuser server environment is the 
retail4/ retailm combination. These Cache ObjectScript routines are designed to demonstrate and give 
practice with the following concepts: multiuser access to a common central database by different kinds of 
processes, designing and using a database to document transaction history, generating unique numbers for 
transactions, object locking and concurrency control. With access to the RMU Cache ObjectScript multiuser 
server, student and faculty can invoke the retail 4 or other routines remotely from different locations 
while conversing on the phone. 
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3.4 Class Notes 

The development of “class notes” for face-to-face meetings was employed in teaching the SVI student. 
Considering the extent of detail in a programming course it seemed prudent to develop a record of 
transactions in a class meeting to give the student detail on what was covered and to assure a way to review 
details prior to a job interview or during employment in the field. It is important to capture the student’s 
problem-solving achievements in the Class Notes. The disadvantage of the Class Notes as carried out is 
likely their uneven quality and lack of consistent organization or of indexing. 


Table 2. Proposed Objectives for Class Notes 

Document challenges and technical problems encountered and describe techniques for solving technical problems, 

especially of managing screen readers and navigating the API and the screen in general. 

Identify and document strategies, habits, and procedures for quality control, such as assuring that potential error 
indications have been routinely checked, making certain where the cursor is on the screen before proceeding, verifying 
program operation, etc. 

Identify and document programming challenges, such as left and right quote marks in code text copied from word- 
processing of web-based sources. 

Document application programming interface (API) limitations discovered as a basis for vendor followup. 

Document relevant and informative links discovered. 

Record interesting and useful examples of code. 

Preparing printed materials for the SVI student required thought and negotiation to assure ease of use. 
Editing and control interfaces encountered in this project were InterSystems Cache Studio and a Linux editor, 
such as vi, and WinSPC. Instruction should cover the pitfalls of copying and inserting text. 


4. LIMITATION, CONCLUSION, AND RECOMMENDATIONS 

The limitation of this study that it is based only on a single set of experiences with a single student must be 
remedied by collaborating with programs at other universities facing similar needs. A plan to develop a 
broader project to obtain more general results has been devised. 

The pervasive use of visually-oriented instructional approaches in web pages and on-line instruction and 
the use of virtual machine software to deliver secured access to all students may ignore the accessibility 
requirements of SVI learners. A systematic approach to SVI accessibility is required to assure opportunity for 
learning in the digital age. Multiple representation modes, as recommended by UDL, is a vital strategy to 
improve accessibility, as the student can then consult feedback from a second representation mode to assess 
the adequacy of an initial mental model. Today’s SVI students rely less on Braille and more on screen 
readers. These students are familiar with and customarily use electronic communication technologies. 
Specific recommendations follow in Table 3. 

Table 3. Recommendations 

Assess the individual SVI student’s competence with literacy (e.g. Braille) and accessibility tools (e.g. screen readers) 
upon admission to <university/college>. Take assessment results into account to plan accommodations for the student. 
Support and enhance the student’s preferred and accustomed modes of access to instructional materials. Have support 
guidance ready and available for each SVI student and instructors. Eaculty should receive guidance information before 

the term begins to the extent feasible. 

Decide on appropriate faculty development to make available for instructors of each SVI student and for all interested 
faculty. Consider including an orientation to SVI accessibility in a faculty meeting. 

Develop guidance for teaching SVI students with lists of resources. Ideas might be gained from the page posted by the 
University of Northern Iowa: http://www.uni.edu/walsh/blindresources.html or from Pennsylvania State University’s 
page: http://accessibility.psu.edu/visuallyimpaired. Encourage accessibility by involving any given student in the 
evaluation of accessibility. Recommend an approach to teaching that incorporates periodic assessment during the 
course term of accessibility of course instructional materials. Require every instructor of an SVI student to consult 

guidance and sign off. 

Review accessibility of online courses, course web pages, and instructional support software (e.g. VMWare) to assure 
compliance with ADA. Involve the local Center for Educational Technology and IT Services. Assure that information 
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and guidance is provided to each online course developer. Use the University of Illinois Function Accessibility 
Evaluator 1.1 (FAE): http://fae.cita.illinois.edu/ and consult the resource “Creating Accessible Web Pages” from the 
University of Colorado: http://www.colorado.edu/webcom/access/, W3C’s “Web Content Accessibility Guidelines” 
(2008) and other such resources as needed. Act in accordance with findings to ensure accessibility. 

Review options for providing audio and/or haptic/tactile diagram or image accessibility. Examples of diagrams: 
Unified Modeling Language (UML), Entity Relationship Diagrams (ERDs), M Programming Global Arrays. 
Examples of image graphics: maps, Geographic Information System (GIS) outputs. Develop procedures and budget 

for making accessibility support available for SVI students. 

Develop procedures for recording class notes for SVI students if desired. Ascertain any given student’s preferences for 
such note-taking. Determine if any given SVI student wishes a “header” symbol for indexing documentation and, if so, 
which one. 
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ABSTRACT 

Collaborative design empowers learning management system (LMS) providers and end users (online students) to develop 
a vibrant teaching and learning community. Successful periodic collaborations utilizing collaborative web tools between 
these two pivotal groups can produce the next series of eLearning platforms that are fertile grounds for successful online 
students. 
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1. INTRODUCTION 

Despite mounting efforts of post-secondary online institutions to increase student retention and completion 
rates, it is troubling to see continuous rise of dropout rates online students leave behind. Online learning is a 
dominating element in this technological society. Several students walk through the doors of virtual 
classrooms, only to find themselves storming out with mounting student loan debts. More than 60 percent of 
two-year colleges offer online courses; the number of students enrolled in online courses dramatically 
increased reaching to over 3 million students (Thiede, 2012). Surmounting attrition rates continue to baffle 
online institutions. Even if students are motivated to learn, several of them face challenging obstacles that 
deter them from succeeding. Only a successful online learner (SOL) can rise above these challenges. How 
does a typical SOL rise above these challenges? What tools does a typical SOL use to succeed? Linally, how 
can LMS designers create a fertile ground of successful online learners into their online learning platforms 
(student portals)? 

Students face feelings of unfamiliarity in their online classrooms, which makes navigating through a new 
eLearning platform too difficult and time-consuming. Consider an adult learner returning to school only to 
face an overwhelming burden of learning to navigate through a student portal while trying to keep up with 
assigned courses. Motivation and self-efficacy depend on how well an adult learner can adapt to a new 
environment. A perception of failure or “difficulty to keep up with the class” can drive a new student to 
abandon aspirations of academic success altogether. 

This paper has three objectives. The first objective is to examine a 2008 study wherein learning 
management system (LMS) designers and end-users (students and instructors) collaborated to identify LMS 
gaps. The second objective is to examine the profile of an online student in the 2L' century. The final 
objective is to construct a conceptual framework for online collaborative sessions between LMS designers 
and online students. This conceptual framework is a contribution to the field of education, particularly, 
distance education, initiating a structure of periodic collaborative sessions between LMS designers and online 
students to create virtual classrooms conducive to learning. This paper demonstrates that comprehensive 
collaborative sessions between LMS designers and their end-users (online students) are crucial in creating a 
vibrant learning and teaching community. 
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2. GAPS IN LMS DESIGNS 

Davoli et al (2008) conducted a qualitative case study wherein LMS designers worked with their end users 
(college students and instructors with different levels of computer skills) to evaluate the writing and 
collaboration features of three well-known eLearning platforms. Participants shared their insights with LMS 
designers and described several technical and aesthetic issues. These issues include system usability, 
compatibility, navigation, and streamlined workflow. The participants evaluated three platforms, (a) Ping 
Pong, (b) Blackboard, and (c) Moodle, addressing issues related to online writing activities (Davoli et al 
2008). The purpose of the study was not to designate which LMS platform is the best virtual learning 
environment, but rather, to inform LMS designers the effectiveness of their web tools in assisting their 
participants to write and communicate with each other (Davoli et al 2008). 

The gaps in LMS designs that Davoli et al (2008) discovered were usability, compatibility, navigation, 
and streamlined workflow. Usability pertains to the straightforwardness and clarity of the system. 
Compatibility pertains to the users’ access to student portals using various browsers and systems. Navigation 
pertains to the (a) number of clicks required for users to access another section, (b) the ability of users to 
view class content without having to scroll all the way down, and (c) the ability of users to understand what 
each link is for, eliminating any type of guesswork. Finally, streamlined workflow pertains to aesthetic issues 
such as small icons too hard to see; fonts too small to read; or links and tabs too ambiguous. 


3. PROFILE OF A 21^^ CENTURY ONLINE LEARNER 

According to Palloff and Pratt (2003), online learners come from diverse age groups, educational 
backgrounds, gender, culture, beliefs, and traditions. Understanding what makes an online learner successful 
can help LMS designers discover what online learners need to persist and succeed in their academic plans. A 
successful online learner (SOL) must have the technological tools and technological knowledge to access the 
online classroom and to keep up with the course work (Palloff & Pratt 2003). The SOL can connect the 
breadth and the depth of knowledge along with prescribed learning outcomes to practical experiences and 
through reflective exercises (Palloff & Pratt 2003). The SOL has the motivation, drive, determination, and 
commitment to keep up with the coursework, to maintain satisfactory grades (at least), and to transform 
knowledge content into practical applications (Palloff & Pratt 2003). Finally, the SOL keeps a regimented 
schedule (Palloff & Pratt 2003) for timely assignment and discussion post submissions. 


4. CONCEPTUAL FRAMEWORK 

Pace (2011) indicated that several online students and their institutions are dissatisfied with their LMS 
providers particularly with systems’ functional features. Welder (2008) indicated that several LMS customers 
are constantly on the lookout for a better LMS provider. The conceptual framework proposed in this study is 
a systematic process in which LMS designers and online students can collaboratively create the next series of 
eLearning platforms. To do so, LMS designers must ask questions related to their systems’ usability, 
compatibility, navigation, and streamlined workflow features and directly align these features against the 
characteristics of an SOL. 

Figure 1 demonstrates the integrative relationships between the four aforementioned LMS gaps with the 
characteristics of SOL. Usability features that contribute to success have the following features: (a) 
straightforward commands and keys, (b) interactive activities and tools to promote mastery, and (c) 
applications that promote organizational skills. Having technological tools that are compatible with one’s 
browser or system can add to one’s sense of commitment and determination to succeed. Compatibility 
solutions include students’ access to technical tools that allow them to thrive in their online classrooms. 
Testing eleaming platforms with the least popular to the most common browsers and systems, including 
mobile devices, can ensure course accessibility. It is easy to assume that all end-users can use a particular 
browser, system, or mobile device. However, online students come from all parts of the world. A browser or 
a system that is widely used in United States may not be accessible to students outside of the western region. 
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Motivation tools in online classrooms including digital badges imbedded in elearning platforms can 
inspire students to complete learner-centered activities and assignments, including capstone projects and 
discussion board participation. Mandatory personal development plans can also increase motivation. 
Reflections on the significant milestones achieved can positively affect one’s commitment and determination 
to persist in an academic program. Enhanced navigational features include tools that can help students 
retrieve previous and relevant lessons, allowing them to gain the necessary depth of knowledge to succeed. 
Simple yet powerful navigational tools attributing to sociocultural interactions with peers and instructors, 
known to promote critical thinking skills based on Vygotsky’s sociocultural theory (Wertsch, 2008), are 
necessary for online students to have to reach higher order thinking. Instructors also need opportunities to 
leave personalized feedback to motivate their students. Acquiring critical thinking skills enables students to 
maintain satisfactory to superb grades. 

Students should have access to practical applications and interactive tools known to improve grammar, 
writing, and research skills. All virtual classrooms should promote streamlined workflow. Students working 
within their eLearning portals should have easy access to outside resources such as research articles, blogs, 
and relevant video clips or graphic presentations, known to eliminate the redundancy of plain lectures. Just as 
professional development plans are critical to help motivate students, interactive timelines demonstrating 
progress reports can also build their confidence. 



and writing 

/ 

Accesses interactive timeline 
to course and degree completion 


Figure 1 . New Elearning Features Collaboratively Designed to Develop a Successful Online Learner. 
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5. COLLABORATION BETWEEN LMS DESIGNERS AND END-USERS 

Collaboration between LMS designers and end-users suggested in this paper include periodic synchronous 
web meetings. Regardless of the web tools used to connect LMS designers and online students, both test 
facilitators (LMS providers and designers) and students (testers) should each provide detailed information 
about the problems, solutions, and issues of the featured eLearning platform related to its usability, 
compatibility, navigation, and streamlined workflow. Online student participants should test these features 
and verbalize or write their opinions or ideas about them, while LMS designers should collect these 
qualitative data. Subsequent collaborative sessions should be scheduled, allowing the online student 
participants to retest the featured eLearning platform. 

Online students in this collaboration must have varying computer proficiencies enabling LMS designers 
to observe their elearning platforms in the hands of three sets of testers: (a) students new to online 
classrooms, (b) students who have 1-3 years of experience in online classrooms, and (c) seasoned online 
students with more than 3 years of experience in an online classroom. Examples of questions and concerns 
that are important to address during these collaborative sessions include: Are there too many clicks to 
perform one activity? How many clicks are too many or too annoying? Are all components visible in one 
screen? Is there enough contrast between the background and the content? Are the links, tabs, or buttons too 
ambiguous? How many clicks does it take to see a grade book, a personal development plan, instructor 
feedback? Does the platform provide video clips and graphics easily accessible and easily managed by 
instructors? Can the dashboard be customizable? Can a student contact an academic advisor or financial aid 
counselor directly from the dashboard? Does the student portal provide organizational tools such as digital 
day planners, reminders, notebooks, sticky notes? Can a student access a “time to completion progress” 
report? Can a student access the school email through outside sources? Are all these features accessible and 
consistent in any mobile device? 


6. CONCLUSION 

The collaborative sessions between LMS designers and online students should be comprehensive, allowing 
both groups to cover all issues and concerns. Future dates must be in place for subsequent collaborative 
sessions. Online student participants must gain opportunities to evaluate the final changes that LMS 
designers implemented. Separate collaborative sessions should be replicable for subsequent meetings 
between LMS designers, instructional specialists, and online instructors. Finally, online student participants, 
instructional designers, and online instructors should receive periodic updates conducted on eLearning 
platforms for review. The conceptual framework described in this paper can benefit all parties. Now is an 
opportune time to harness technology, create, improve, and enhance features to produce an abundant 
generation of successful online learners. 
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ABSTRACT 

Adopting the multiliteracy concept and embracing the challenge of developing meaningful and captivating classes for 
Portuguese as Eoreign Language in Brazil, this paper proposes an approach which includes the use of different 
technologies to learn and teach Portuguese, the reading of graphic novel adaptations of Brazilian literature classics and 
the creation of a multimedia product called book trailer based on those novels. The whole process aims to promote aware, 
committed and multiliterate students. 
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1. INTRODUCTION 

It has been a challenge to provide meaningful, relevant and interesting content for heterogeneous students of 
Portuguese as Foreign Language in Brazil. The challenge becomes larger if we consider the changes society 
has been facing because of the shift from a page-base to a screen-base world. This change-in-progress has 
been affecting many aspects of our society, for example, the form “new communications media are reshaping 
the way we use language” (Cope and Kalantzis, 2000:06), raising questions as “how do these changes affect 
the ways we learn, use and teach languages?” and debating issues as “the status of CALL [computer-assisted 
language learning], its theoretical grounding, its cultural embeddedness, and its effectiveness” (Kern, 
2006:183). This paper aims to develop an approach which considers the use of different technologies (once 
computers have become a convergent device and there has been a growing access to other technologies as 
cameras, camcorders and free software), the importance of cultural aspects during the learning process and 
the rule and effectiveness of technology in the educational field. About the last one, as Kern stated, it “is not 
technology per se that affects the learning of language and culture but the particular uses of technology” 
(Kern, 2006:183). Agreeing with Kern, some specific questions will guide this research as: (a) will the use of 
a multimodal approach help the students to understand linguistic and cultural aspects of Brazil? Will the 
technology adopted be an obstacle or will it make the attempt to work with Brazilian texts easier? Will the 
learning process be more meaningful? Will the students be more committed with the new kind of activity 
which will be suggested? How can the multiliteracy concept be worked in a second language class through 
different technologies? 


2. MULTILITERACIES 

One consequence of those changes mentioned above in relation with the educational system is that students 
are demanded to be capable of reading and producing multimedia texts as well as the “old” printed ones. 
Students are expected to have multiple literacies and educators are expected to teach students to become a 
multiliterate person. As Hicks (2006:3) wrote, the discussion about how students “can produce and consume 
texts in critical, creative and ethical way” has become “crucial in creating an informed, engaged and 
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multiliterate citizenship”. Multiliteracy is a concept which presents worthy perspectives when we search 
answers for how literacy teaching can enable students for this switching world, once it “focuses on modes of 
representation much broader than language alone” (Cope and Kalantzis, 2000, p.05) and so it allows readers 
to be “cognitively and socially literate with” a variety of texts, “a problem solver and strategic thinker” 
(Anstey and Bull, 2006:41). 


Table 1. Multiliteracy (Anstey and Bull, 2006:41) 


Tlte multiliterate person can 
inteipret, use, and produce 

'i' 

electronic, live andpaper texts thatenqiloy 
linguistic , visual, auditory, gestural and 
spatial semiotic systems 

4 . 

fiir 

4 ^ 

social, cultural, political, civic and 
economic puiposes 

4 . 

in 

socially and culturally diverse contexts 


Adopting multiliteracy while developing materials for Portuguese as Foreign Language in Brazil sets 
another challenge: the creation of an approach which includes and works with a variety of modes and media, 
promoting critical thinking and enabling students to learn Portuguese in an involving and motivating way. 
Embracing the multiliteracy, considering the profile of the students who will take part in this pilot study, the 
facilities of the center where those students will study this semester, the language syllabus expected to be 
taught and the current publishing industry in Brazil, book trailers and graphic novel adaptations of classics of 
Brazilian literature are potential resources to develop a teaching approach. 

Book trailers have become popular in Brazil (e.g. BookTrailer - http://www.booktrailer.com.br/ 
Home.html), mainly among teenagers and young adults. A book trailer is normally an advertisement for 
novels created by book publishers or authors to sell their books. It can be defined as a multimodal text in 
which a variety of semiotic modes are combined. The multimodality can be explained as an approach that 
understands “communication and representation to be more than about language, and which attends to the 
full range of communication forms people use - image, gesture, gaze, posture, and so on - and the 
relationships between them”(Jewitt, 2009:14). Using a book trailer in a pedagogical approach creates 
“opportunities for students to re-enact the main storyline of books they read, using live video clips, pictures, 
music, voice-overs and other digital tools” (Digital Booktalk). The idea of using a book trailer to work the 
understanding of classics of Brazilian literature is partially based on the “composing” concept fostered by the 
Conference on English Education Belief Statements About Technology (Swenson et ah, 2005). According to 
it, when students create their own product (a video clip, a music, a movie trailer and etc.), their personal 
concept of literacy is reinforced. However, as there are time and some technological limitations to produce 
something very time-consuming (as movie trailers and video clips), the book trailer shows itself as an ideal 
option. The time limitation, in turn, demands students to focus on the main points of a book or a graphic 
novel, exercising selection and condensation of the text, while they are making the book trailer. 

About the graphic novels, they are characterized as “fiction or nonfiction books presented in comic book 
format that require multimodal literacy for understanding” (Hammond, 2009: iii) and they are considered “an 
emerging new literature of our times” (Campbell, 2007, p. 13). As Downey (2009:181) mentioned “graphic 
novels today are being used increasingly by educators to engage reluctant readers, reach out to visual 
learners, and illustrate social and cultural themes and topics”. They are also a flourishing market in Brazil 
which has produced award-winning literary adaptations in the last years. 

The next topic will detail how an approach has been developed including the pedagogical use of graphic 
novel adaptations of Brazilian literature classics, the production of book trailers, and the use of different 
kinds of technologies. 
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2.1 Methodology 

The purposes of this approach are to enable students to (a) use spoken, written, and visual language to 
express themselves successfully to a wide range of audiences and for different goals; (b) employ their 
knowledge of language structure and conventions, as well as media resources and technological tools to 
produce, evaluate, and discuss different types of texts; (c) explore a variety of technological and information 
resources, in a critical and ethical way, to obtain the information and tools they need, creating and sharing 
knowledge. This project also expects students to show their understanding of graphic novel adaptations of 
Brazilian literature classics by creating a multimedia product called book trailer. During this process they will 
also (a) learn film-making and technological vocabulary in Portuguese; (b) discuss the cultural aspects 
spotted in the graphic novels; (c) question the status quo of the society shown in the books, trying to link it 
with our social structure, (d) learn how to write a script; (e) find out how to produce a book trailer; (f) debate 
the pro and cons of copyright and the use of materials available on the internet; and (g) promote collaborative 
and peer work. Adopting this approach we will be able to deal with linguistic and cultural dimensions, as 
well as to the critical use of technological and information resources. 

The participants will be twenty Spanish speaking students from different nationalities, age and gender 
who are in Brazil taking Portuguese classes (intermediate level) at Centro de Ensino da Linguagem 
(Language Teaching Center), which belongs to University of Campinas 
(http://www.unicamp.br/unicamp/?language=en). This center has a multilingual library, computer and 
language lab, and different types of monitors to help the students with linguistic and technological doubts, so 
its infrastructure and facilities enable the development of this project. Also, during the role process, the 
students will have the support of their Portuguese teacher. 

2.2 Sessions 

In the beginning of the first meeting the professor will apply a questionnaire (detailed below) which will be 
used to assess the project. After the students finish answering it, the professor will present the steps of the 
project: (a) the students will organize themselves in groups of interests; (b) the professor will explain how the 
students will show their understanding of the book - instead of writing a traditinal abstract, they will produce 
a book trailer. The educator will define what a “book trailer” is, show some examples, discussing their 
characteristics, the resources applied and etc.; (c) the groups will assess each group's book trailer through a 
check list; (d) the students will decide if they want to show their books trailers through the creation of a blog 
or a channel on YouTube; (e) at the end of this first session, the educator will organize the schedule (both 
face-to-face and online meetings) of the project. The students will have a month to read the graphic novel and 
one month to produce the book trailer. 

During the first month an online forum will be created so the students will have a virtual place to post 
their doubts while reading the graphic novels. After that, a second face-to-face meeting will be attended when 
each group will clarify doubts they still may have about, for example, vocabulary and grammar. During this 
session: (a) cultural aspects shown in the books will be highlighted and analyzed; (b) they will debate how 
culture and society involve a game of power which tries to keep the status quo (in the past and nowadays) 
and how this game affects our lives; (c) an example of a book trailer project plot planner and story board will 
be analyzed, enabling the students to start the first draft of the script; (d) the students will discuss what kind 
of check list they will adopt to evaluate each other’s works; (e) the groups will decide which movie editing 
software they will use - it can vary from one group to other, (f) some tutorials about how to create a book 
trailer will be introduced; (g) the students will have a discussion about copyright and appropriation of content 
from the internet. 

The next meetings will occur online, both through the use of the forum as well as through chats scheduled 
according to the students needs. Before the end of the project one face-to-face meeting will be booked with 
each group. During that session the students will show just for the educator their book trailer, so if there are 
any retouches or corrections, the students will have enough time to do it before their presentation. 

The last meeting will be the one the groups will show their book trailers to their classmates. During that 
session the students will: (a) answer another questionnaire (detailed below) focused on their experience of 
reading the graphic novels and making the book trailers; (b) assess each other’s works. 

2.3 Assessment 

There will be two kinds of assessment: (a) of the project and (b) of the students. The project assessment will 
be based on questionnaires answered by the students (one in the beginning and the other at the end of the 
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project), meetings observation and analysis of the forum and chats. The questionnaires will have open-ended 
and closed questions. The first one intends to observe (a) what kind of experience the students have with 
technology; (b) what they know about Brazilian literature; (c) if they are used to reading graphic novels or 
comic books (and in what languages) and what expectations they have of the course. The second 
questionnaire will focus on their experience during the project and it will be built aiming to answer the 
questions mentioned in the introduction of this paper. Also, it will be partially developed based on the 
answers given in the first questionnaire and on observations made by the educator through the process. The 
student assessment will be split in two. In the first one the students will evaluate each other’s works through a 
check list. The second type of assessment will be the educator’s - she will also use a check list, but an 
expanded one, which will include items as (a) proper use of vocabulary; (b) grammar; (c) pronunciation and 
etc. The results will be shown during CELDA 2013. 

3. CONCLUSION 

Currently students are expected to be multiliterate, to be capable of apply their skills to deal with various 
texts and media. Helping the students to develop those skills (ability to analyze, evaluate, and synthesize 
various forms of text) is a challenge teachers and professors have to face everyday. But we need to be aware 
the “multi-” in “multiliterate” stands for “having many” (Cambridge International Dictionary of English, 
2000:928), something that is not single, homogeneous. So, one multiliterate student will probably present not 
only a variety of literacy, but also different levels of literacies. If we consider a class with more than twenty 
students from different countries, cultures, religions, age, gender and so on, the previous statement about the 
prefix “multi-”- that can be seen as naive or obvious in a first impression - shows its depth. How can 
educators deal with so many variants, with a so heterogeneous context? Bearing that in mind, this project 
intends to show that combining technology (video making), popular culture (graphic novels) and traditional 
academic tasks (reading classic literature) with collaborative work, discussions and an open-mind behavior 
can be an enticing approach to fulfill an impressive challenge. 
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ABSTRACT 

The literature affirms that widespread lapses in corporate social responsibility obligations (unethical behaviors) have 
periodically brought about extensive forfeitures of economic wealth and countless job losses leaving the world economy 
in recession or depression. Put forth as a resolution to unemployment issues the academic literature champions the idea of 
life-long learning as a way to prepare our workforce for the changing nature of occupations. The purpose of this work-in- 
process report is to provide insight into the progress of my research for advocating an inclusion of social responsibility 
obligations into life-long learning agendas. It is my belief that inclusion will advance life-long identification and 
edification of higher social responsibility obligations for all levels of society. 


KEYWORDS 

Life-long learning; ethics; social responsibility 


1. INTRODUCTION 

We are all familiar with the ethical scandals that brought Enron to its knees. But most of the world does not 
know that “the Chernobyl power plant disaster resulted from a meltdown of ethics by Soviet engineers” 
(Jadwin 2006 :72). While Enron brought financial ruin to thousands, Chernobyl brought death, disease, and 
displacement to countless thousands for years to come. 

Moreover, findings from my research of the academic literature clearly support lapses in social 
responsibility obligations (unethical behaviors) by some of our leading financial firms and related industries 
led to the latest downturn of economic activity. As a result of this phenomenon the US and the world 
economy has faced substantial capital depreciation, significant job losses, high unemployment rates, and 
economic instability (Pacitti 2011). Consequently, the literature champions the idea of sustaining cultures 
which support life-long learning as part of a broader global economic solution (Young 2001). Organizations 
like the International Association for the Development of the Information Society (lADIS) and its 
International Cognition and Exploratory Learning in the Digital Age (CELDA) 2013 conference are working 
to define the future of life-long learning. As stewards of this process we must foster life-long learning 
environments that will include an awareness of social responsibility obligations. It is with this thought I 
recognize lADIS’s CELDA 2013 conference as an opportunity to share my research. 


2. PURPOSE 

This paper outlines my research progress on the concept of life-long learning as the vehicle which supports 
both organizational and individual social responsibility obligations (ethics). This genre of research is 
important because it raises awareness and provides guidance for shaping cultures of life-long learning for 
social responsibility obligations. I report that my research efforts are primarily in the literature review stage. I 
have gathered over 500 documents written by academics from all over the world using EBSCOHOST 
(licensed through the University of North Texas). My efforts have afforded me articles relating to human 
resource development (field), ethics (problem), knowledge management (solution) and life-long learning 
(solution). Erom this work and future studies we can ascertain the importance of life-long learning 
experiences and supporting cultures for guiding individuals and organizations though the life-long ethically 
charged minefield. The following sections explain some of the main concepts from the reviewed literature. 
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2.1 Society’s Major Challenge 

Collins (2008:614) emphasizes that “life-long learning has emerged as one of the major challenges for the 
worldwide knowledge society of the future.” An educated and well trained agile work force is essential to 
economic success. The same can be said for fulfilling life-long social responsibility obligations. As 
technologies and events change, human beings will be faced with serious choices involving both technical 
competencies and ethical decisions. It is self-evident that as the US becomes deeply entrenched into the 
global economy, and information and technologies race to levels far exceeding our comprehension, a societal 
need for the fulfillment of social responsibility obligations and the embracement of life-long learning are 
necessary for our social and economic survival. No society, even a highly educated society can prosper 
without a sense of moral duty and social responsibility. 

2.2 Life-Long Learning 

Life-long learning needs to be differentiated from life-long education and job preparation activities. 
According to Galbraith (1995:8) “life-long learning are those changes in one consciousness that take place 
throughout one’s life span which results in an active and progressive process of comprehending intellectual, 
societal, and personal changes which confront each individual”. With this definition in mind, it is easy to 
understand the importance of rejuvenating our awareness and understanding of social responsibilities. 
Galbraith (1995) further contends that life-long education is a process of both deliberate and unintentional 
opportunities that assist and encourage life-long learning. It is conferences like CELDA that provide the 
opportunities to educate it members about social responsibility obligations as well and encourage their work 
in establishing these connections to life-long learning projects. 

While governments cannot be expected to solve all facets of organizational and individual social 
responsibility obligations, governments are influential components of life-long learning. In that context, my 
research confirms the governments all over the world understand the important of life-long learning. The US, 
EU, Australia and other developing countries have legislated life-long learning programs (Youngs, Ohsako, 
Medel-Anonuevo, & United Nations Educational, S. n. (Germany). Inst, for Education 2001; Watson 1999). 
The US in particular has a series of legislated acts: the Higher Education Act of 1998, 2010, and 2011; the 
No Child Left Behind Act of 2001; and the Life-long Learning Act of 2008 (Young, Grant, Montbriand & 
Therriault 2002; 107th Congress 2001; Scott, United States & United States 2010; United States 201 1) 

Most notable of the acts is the Life-long Learning Act of 2008 which specifically instructs the Internal 
Revenue Service to modify the Internal Revenue Code of 1986 establishing citizen life-long learning 
accounts as a funding incentive for citizens to take advantage of life-long learning opportunities (H.R.6036 
2008). Our citizens can contribute earned income into a tax-free savings account to be used for continuing 
education. The United States has emphatically demonstrated its support for life-long learning. While this 
program, by its nature, provides support for people with disposable income at a level where income taxes are 
weighty, it is a significant step in establishing a mentality that life-long learning is important and provides a 
vehicle for its implementation. Tax policy has long been used to influence the behavior of its citizens. The 
deduction for interest costs on a mortgage payment has long promoted a culture of home ownership. Thus, it 
follows there is potential for life-long learning accounts encouraging a culture of life-long learning. 

Lest we feel that it is not our responsibility to promote social responsibility awareness within a life-long 
learning context, we need to examine what the US government legislates and take up partnership with their 
ideas and efforts. 

2.3 Social Responsibility Obligations (Ethics) 

Much like the topic of life-long learning there is much research in the US, Europe, Australia, and other 
developed countries on the broad topic of corporate social responsibly. 

According to Matten and Moon (2004) corporate social responsibly is an umbrella topic that includes 
business and social relations as well as business ethics. Eor my research I broaden the topic even more to an 
organizational level that includes business, government, and other forms of organizations or agencies that are 
comprised of individuals. While organizational social responsibility obligations are often formulated from 
laws, regulations, and accepted best practices, ultimately they are executed in an environment based upon the 
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culmination of individuals’ collective life’s experiences. Thus, individuals with their learned morals and 
personally developed value systems are the deciding factor for all organizational social responsibility 
obligations. However, as individuals, what we learn as a child is reshaped over and over again as we 
encounter our educational culture, our social culture, then onto our employment culture. While there is much 
debate about “nature verses nurture” across every social behavior theory much research extends the theory 
that morals and values are learned concepts (Bronfenbrenner & Ceci 1994; Bors 1994; Doring 2010). My 
interest in this research stems from my personal family experiences. If I may share a personal story that many 
of us can relate to: 

“I personally learned many of my values and resulting personal ethical behaviors from having parents 
who promoted good family ethical behaviors. I can recount a single experience that shaped many of my 
values and my resulting personal ethical behavior more than anything else. When I was about six years old, 
while shopping with my mother, I stole an animal-shaped rubber eraser. When I exhibited my treasure to my 
mother, she explained that to take something that was not my property was stealing and wrong. She further 
added that the eraser was the store owner’s property and would miss the benefit of owning it. She had me 
return it to the store owner with my apology. The mental image that most impressed me was that the store 
owner was a person. I understood I had hurt or had the potential to hurt that person. It was a lesson learned 
that is still vivid in my memory. My personal story emphasizes that our own personal ethical behaviors are 
built upon a combination of learned values from our experiences in combination with an ability to see beyond 
our own self-interests” (Mayes 2013:1). 

My experiences and the literature suggest that over a lifetime humans build a value bank which they will 
use to draw upon when faced with ethical decisions (Doring 2010). Moreover, individuals feel better when 
they comply with what they have learned from personal experiences. According to research, humans also 
build an empathy bank created from positive personal experiences. When we understand how other people 
feel and perceive their reality we find our ethical behavior in line with what others would expect from us 
(Kasser, Koestner & Lekes 2002). Whereas, organizations can establish well defined value banks they 
cannot possess empathy per se. However, their leadership and members collectively can. It is not an 
overstatement to say that the organization’s culture (people) provides both formal and informal ethical 
guidance that provides a constant reinforcing to the value and empathy banks. Thus, reinforcement of social 
responsibility obligations is an important part of any life-long learning culture. 


4. CONCLUSION 

LeClair & Ferrell (2000) posits that an understanding of our values is important to our self-efficacy regarding 
ethical behaviors. For that reason, I believe that cultures of life-long learning must to be created and nurtured 
in order to propagate ethical behaviors by our citizenship. My research is based upon the challenge that if our 
society supports cultures of life-long learning it must include effective methods that edify understandings and 
self-awareness of social responsibility obligations. 

With organizations like lADIS working to define the future of life-long learning via the latest advances in 
supporting technologies, they must foster life-long learning environments that will include ongoing 
awareness of social responsibility obligations. Lest we want to abdicate our social responsibility to a few 
college courses on business ethics, in a study Claudot, Alla, Ducrocq & Coudane (2007) conducted; they 
concluded that learners do not attribute any value to ethics being taught in lecture class settings. This is a real 
opportunity for lADIS. Can on-line education and training technologies; and social software succeed where 
lectures fail? 

Our society understands that ethical behaviors are not skills attained overnight. Still and all, while the size 
of this task seems daunting and limitations are endless, success can be added one concept at a time. By 
integrating social responsibility obligations into the culture of life-long learning, only then can we expect 
successful outcomes. Our society and association only need to show incremental improvements for fostering 
ethical behaviors. What we learn from our experiences and research can be used by all those involved in life- 
long learning fields. Government, educators, and businesses can use this knowledge in programs that 
contribute to the cultures of life-long learning. In the end, maybe we will say “Life-long Learning and Social 
Responsibility Obligations” in the same sentence. 
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ABSTRACT 

We have been developing a digital concept maps website (www.conceptmapsforleaming.com) based on the principles of 
effective assessment for learning. The purpose of this paper is to reveal its promising contributions to formative 
evaluation practices. The website reduces the workload of teachers as well as provides immediate and delayed feedback 
on the weaknesses of students in different forms such as graphical and multimedia. For the following study, we will 
examine whether these promising contributions to assessment for learning are valid in a variety of subjects. 
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1. INTRODUCTION 

Assessment is one of the crucial components of education (Gikandi et al, 2011), and is required for three 
broader aims (Black, 1993): 

• The certification of individual student achievement; 

• The accountability of educational institutions via the comparison of results; 

• Direct assistance to learning through useful feedback. 

To accomplish one or more of these purposes, assessments can be designed as either summative or 
formative. While the main purpose of summative assessment is to categorize students’ performance by 
assigning grades, the basic aim of formative assessment is to identify students’ specific strengths and 
weaknesses in order to facilitate further learning (Cizek, 2010). Assessment for learning is to assess student 
performance and provide feedback during the process as well as to act on the provided feedback in a way that 
benefits that student’s learning (Trumpower & Sarwar, 2010; Filiz et al, 2012). 

1.1 The Effectiveness of Assessment for Learning 

Despite the challenges to assessment for learning including resources and time (Cizek, 2010), several studies 
reveal the positive impact of formative evaluation on students’ achievement. For example, Black and Wiliam 
(1998) concluded that assessment for learning increases students’ performance after reviewing around 250 
articles based on formative evaluation. These studies were related to feedback, self-assessment, and peer 
assessment. Likewise, Nyquist (2003) came to a similar conclusion after conducting a meta-analysis on use 
of feedback for formative assessment purposes. And, in an empirical study of 24 teachers who received six- 
months of training to develop formative assessment practices, William et al (2004) demonstrated the positive 
impact of using assessment for learning in classrooms. 

However, the generalizability of these findings has been questioned by some. For instance, Bennett 
(2011) has argued that the mean effect size computed by Black and Wiliam (1998) is based on studies that 
are too diverse to be meaningfully combined, and that the meta-analysis conducted by Nyquist (2003) was 
too narrowly focused on the college-level population. Similarly, Dunn and Mulvenon (2009) have raised 
methodological concerns that may limit the conclusions drawn by Wiliam, et al. (2004). Because of these 
issues, the specific principles of effective assessment for learning should be further addressed. 
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1.2 The Principles of Effective Assessment for Learning 

Tmmpower and Sarwar (2010) have condensed the criteria of effective formative evaluation into four 
necessary conditions in the context of technology assisted assessment for learning. 

Assesses Higher Order Knowledge: The formative evaluation task must allow teachers to assess higher 
order knowledge of students (Tmmpower Sarwar, 2010). Perkins (1993) notes that lack of higher order 
knowledge causes misconceptions. Therefore, assessing higher order knowledge is one of the cmcial 
functions of any computer based formative evaluation software. Tmmpower et al (in press) noted that 
concept mapping tasks assess higher order knowledge. 

Identifies Specific Strengths and Weaknesses: The formative evaluation task must identify specific 
strengths and weaknesses of students’ knowledge (Tmmpower & Sarwar, 2010; Cizek, 2010). There are 
several ways of doing this. For example, an expert concept map might be given to students to compare with 
their concept map (Tmmpower et al, in press). 

Provides Useful Feedback: The assessment for learning task must provide useful formative feedback on 
students’ weaknesses. Shute (2008) reviewed research on formative feedback and identified several criteria 
of effective feedback, including that it should be simple, objective, provided in a variety of formats other than 
text, delivered in a timely manner, and give clear suggestions on how to improve. 

But, in order for these principles to be effective, students should also be required to reflect on the 
formative feedback (Fontana & Fernandes, 1994; Frederickson White, 1997; Boston 2002), and be given 
opportunities to use it to modify their previous work (i.e., revise and resubmit assignments/test answers). 

Is User Friendly: A majority of teachers think that formative feedback is a difficult and time-consuming 
task (NRC, 1999). Accordingly Tmmpower et al (in press), suggest that technology assisted assessment for 
learning applications should include automated assessment, evaluation, and feedback mechanisms. In 
addition, these applications should allow students to monitor their performance. 


2. BRIEF PRESENTATION OF CONCEPT MAPS FOR LEARNING 
WEBSITE 

In our research lab, we have been developing a concept mapping website (conceptmapsforlearning.com) 
based on the aforementioned principles of effective formative assessment (Filiz et al, 2012). Within the 
website, students create concept maps on a particular subject and then receive individualized feedback and 
associated instructional material (e.g., videos, website links, examples, problems, etc.) based on a comparison 
of their concept map and a subject matter expert’s map. After students study the feedback and instructional 
material, teachers can track their progress by having them create revised concept maps. 

Expert concept maps on a variety of topics (e.g., statistics) are stored in a repository within the website. 
Teachers using the website can choose from amongst the topics. Students are then provided with the concepts 
corresponding to the chosen topic and rate the degree of relationship between the concepts in order to 
generate their concept map. The website then compares eaeh student’s concept map with the expert concept 
map to generate individualized feedback for each student. Feedback is comprised of a visual presentation of 
the expert concept map superimposed over the student’s with any discrepancies highlighted by different types 
of links. As seen in Figure 1, a black line appears provided that there is a link between two concepts in both 
the expert’s map and the student map (i.e., a relevant link). A grey dotted line appears if there is a link 
between two concepts in the student map, but not in the expert’s map (i.e., an extraneous link). Finally, a red 
dashed line appears if there is a link between two concepts in the expert map, but there is no link between 
these concepts in the student map (i.e., a missing link). 

In addition to this visual feedback, additional instructional material is provided for any missing links. 
When students move the mouse cursor over a missing link, a text message appears which explains how the 
associated concepts are related; these explanations have been provided by subject matter experts, but can be 
modified by individual teachers using the website. Further, if students double click on a missing link, they are 
able to access linked instructional material intended to illustrate the ways in which the associated concepts 
are related (e.g., videos, website links, examples, problems, etc.); again, this material has been provided by 
subject matter experts, but additional material can be added by individual teachers (Figure 1). 
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Eigure 1 . An example of a student's feedback map and an associated linked material 


3. HOW DIGITAL CONCEPT MAPPING WEBSITE CONTRIBUTES TO 
ASSESSMENT EOR LEARNING 

One of the crucial contributions of the concept maps for learning website to formative evaluation is to 
automatically identify each student’s misunderstandings and misconceptions. Typically, the task of 
diagnosing misunderstandings is a difficult one, requiring both specific skills (e.g. how to conduct an 
interview) and plenty of time for both developing an appropriate assessment task and then for evaluating it 
(Browning & Lehman, 1988). Conversely, teachers are required to provide very minimal input in the concept 
maps for learning website. Although they may add their own explanations and linked content, they are only 
required to choose a topic and submit a list of students for whom they wish to grant access to the website. 

Moreover, providing feedback on the weaknesses of students in different forms is another of the crucial 
contributions of the concept maps for learning website to formative evaluation, for research has shown that 
students are most likely to ignore verbal and written feedback (Shute, 2008; Lee, 2009). The concept maps 
for learning website provides both visual feedback and linked feedback as a form of associated learning 
activities including videos, games, or cartoon. 

An additional contribution of the concept maps for learning website is that it is capable of providing 
immediate feedback. Shute (2008) note that although high-achieving students may benefit from delayed 
feedback, immediate feedback might be more useful for low-achieving students. The author also reveals that 
immediate feedback is required for difficult tasks which are associated with higher order knowledge. 
Therefore, any technology assisted formative evaluation tool must provide immediate feedback. 

Furthermore, the concept maps for learning website is most likely to promote equitable education 
(Gikandi et al, 2011), because each student’s weaknesses (misconceptions and/or misunderstanding) and 
strengths is diagnosed through computer based formative evaluation software. In addition, students are able 
to study received feedback from a variety of associated instructional multimedia materials. 

Finally, our concept maps for learning website may help students improve their general problem solving 
skills. In a related study, Schacter et al (1999) found that computer based concept mapping tasks improve 
students’ problem solving performance. 
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4. CONCLUSION 

The aim of this paper was to examine how our concept mapping website contributes to assessment for 
learning practices. Briefly, these contributions are as follows: (1) Assessing students’ higher order 
knowledge, (2) Identifying each student’s misunderstandings and misconceptions, (3) Providing delayed and 
immediate feedback on the weaknesses of students in different forms, (4) Promoting equitable education, (5) 
Improving problem solving skills 

For future studies, we are planning to create more concept maps related to different subjects. Therefore, 
whether these promising contributions to assessment for learning are valid in different subjects will be 
examined. 
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ABSTRACT 

Millions of hours are spent on assignments in university education every year. This exploratory paper describes ways in 
which instructors can create assignments that not only educate students, hut also create lasting online contrihutions for 
use hy scholars and future students. I describe case studies of assignment types I have used effectively: Paper Summaries, 
Contributions to Wikibooks, Creation of Mnemonics for a Wiki, Online Elash Cards, and Actual Research. Although 
formal study has yet to be conducted on these methods, in addition to making external contributions these assignments 
appear to have the benefits of higher motivation and greater exposure to scholarly research. 
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1. INTRODUCTION 

As of 2011 there were 23.8 million college and graduate students in the United States alone (U.S. Census 
Bureau, 2012). Every year, these students work on millions of assignments, and instructors and teaching 
assistants spend millions of hours grading them. The vast majority of these assignments help the students 
learn, but do no good for anybody else. The products of these assignments are usually discarded. This is an 
enormous waste of resources. I argue that not only should assignments help students learn, but they should 
also benefit the wider educational and research communities. 

In this short paper I will describe a number of assignment types I have introduced over the past years that 
I believe 1) facilitate learning, 2) are particularly motivating, and 3) contribute to the greater educational and 
research communities. Although this chapter will focus on my field, cognitive science and artificial 
intelligence, with some creativity most of the methods will translate to other fields in university education. 


2. ASSIGNMENT TYPES 

I will describe five project types that I have used effectively: Paper Summaries, Contributions to Wikibooks, 
Creation of Mnemonics for a Wiki, Online Flash Cards, and Actual Research. 

2.1 Paper Summaries 

For classes with fewer than forty students, I assign the creation of a paper summary. When they are turned in 
the summaries are graded, edited, and put on a website I maintain (Davies, 1999). The students must 
summarize a piece of scholarly work that has not yet been summarized on the website. 


389 



ISBN: 978-989-8533-18-0 © 2013 lADIS 


The summary is supposed to include the basic claims of the paper, and the evidence or arguments for 
those claims. The students are given a standard template to use so summaries include similar kinds of 
information. First is complete citation information, in two formats, the American Psychological Association 
(APA) and BibTeX (for users of the LaTeX typesetting language used in computer science and other 
technical fields.) This makes it easy for other researchers to copy and paste the reference into their own 
papers. 

Next comes the name of the author of the summary, and his or her permanent email address. I tell my 
students that unless they say otherwise, their assignments will be put on the website, and if they don’t want 
their names on it, I will publish the summary author name as “anonymous.” Approximately 5% of students 
wish to be anonymous. 

I also require the students to include a list of specific things one could cite the paper for. For example, in 
the summary (Davies, 2000) of Larry Barsalou’s paper on Perceptual Symbol Systems, the statement 
“Amodal symbols are redundant if they just link to the percepts” is in the “cite this paper for” section. It is for 
claims, argument conclusions, original ideas, names of software systems reported, quotable wording, etc. The 
motivation for this is search: if you read a fact or claim but cannot remember where you read it, a web search 
for that fact might turn up the summary, allowing you to cite the paper. 

Finally, there is the summary itself, which I allow the students to structure any way they see fit, except 
that I ask them to associate page numbers with statements, so that readers of the summary can easily find 
what's being summarized in the original paper. 

Approximately 95% of the summaries I have collected in this manner have been of high enough quality to 
put on the site, mostly with only minor alterations — usually formatting. 

This kind of assignment has several educational advantages. First, because students find their own paper 
to summarize, they get experience looking through journals, giving some idea of the state of the art. Second, 
they read several abstracts, and finally choose a paper they are really interested in, which is motivating. 
Third, they get exposure to real research, reading non-textbook science and understanding an actual scientific 
paper, which many second-year undergraduates, for example, have never done. Finally, knowing that their 
work will be on the web is further motivation to do a good job. 

Since all fields have scholarly papers, this method applies to any discipline. 

2.2 Wikibooks 

The Wikimedia foundation, which manages the Wikipedia, also has a series of wikis called “Wikibooks,” 
which are for the creation of free content textbooks that anyone can edit from a web browser. I require 
students in my artificial intelligence classes to write chapters or chapter sections for the Artificial Intelligence 
Wikibook (Wikimedia Foundation, 2009a). One year I assigned each student to write a piece about a search 
strategy that had not already been covered in the Wikibook. Perhaps, in ten years or so, the book will be 
sufficiently mature so that my AI students will not need to purchase a textbook at all. 

As with the summaries, any field can use this method, contributing to (or starting) an online textbook. 

2.3 Creation of Mnemonics for a Wiki 

One of the most difficult parts of cognitive science education is the memorization of brain areas, in terms of 
location, name, and function. Most students learn these things through repetitive drilling of the information, 
rather than using mnemonics, which have proven to be very effective for memorization. Unfortunately, text 
books and teachers rarely give students mnemonics to use. Because creating mnemonics requires both 
knowledge of their effectiveness and a good amount of effort, they are rarely created by students on their 
own. 

For each fact that needs to be memorized, however, the whole world only needs a single good mnemonic. 
The famous "Roy G. Biv" helps everyone remember the colors in the spectrum— it is not that each person 
needs to create his or her own mnemonic for the colors. 

This is the motivation behind the Brain Areas Mnemonics Wiki project (Davies, 2009). The wiki is a 
place where one can find mnemonics for remembering what brain areas are associated with what functions. 
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Students are required to look at the wiki and see which brain areas have not been addressed, find three 
unaddressed brain areas, and create mnemonics for remembering the functions in which those areas are 
implicated. They present these in class, and as a group we improve them before publishing them to the web. 
A class of 15 students will create 45 mnemonic devices in a single semester. 

One student created a textual mnemonic for the association of the Basal Ganglia with motor control, 
cognition, emotions, and learning. The mnemonic was this: Imagine a person trying to learn to dance, 
unsuccessfully, next to a bee hive. The sudden movements make the Bee Gang (Basal Ganglia) angry and 
decide to attack. 

Anyone can start a wiki, free, with Peanut Butter Wiki*. 

2.4 Online Flash Cards 

Anki is a member of a family of programs that implement spaced learning in an electronic flash card format. 
Anki, the open source system I use, is a program designed to be used every day. The key point is that the 
software, rather than the user, decides which cards are to be reviewed each day. The software keeps track of 
which facts you got right and wrong to determine how long you should wait before reviewing that fact again. 
The idea is that the best time to review a fact is just before you're likely to forget it. So if you get a flash card 
correct, it might present it to you again in two days, and if you get it correct again it will present four days 
from then, then eight days, etc. The problem with traditional flash cards is that you waste a great deal of time 
reviewing flash cards you already know very well. 

Certain domains require a good deal of memorization (e.g., medicine, biological sciences, foreign 
languages, law), and programs like this can be of enormous value. With Anki, users can create decks of cards 
and share them with other users. Any time one wants to remember a fact, one can type it into the program in 
a question and answer format. 

In my assignment, I asked each student to pick one lecture from the class and to make a deck that covered 
all of the factual information from that lecture. In a single year all of my classes had on-line flash card decks 
for all lectures. 

Like mnemonics, the flash cards only need to be created once for everyone to benefit. Anyone can 
download decks of cards to memorize the facts therein. After only two years of teaching and assigning the 
creation of cards, all the notes from lectures and readings were in flash card form for future students to use. 
Since creating Anki cards does not require any computer science-specific knowledge, this assignment can be 
used in any discipline. 

A final benefit of this kind of assignment is that students are introduced to programs like Anki, which can 
help them in their schoolwork in general. 

2.5 Actual Research 

Finally, students can be assigned to conduct actual scientific research as a class assignment. The feasibility of 
this method varies greatly from discipline to discipline. In high-energy physics, for example, being able to do 
new scientific research requires years of graduate training and very expensive equipment. In contrast, for 
artificial intelligence it's relatively easy, since there are a great many problems that have never been 
addressed by anyone. 

Any project that requires computer programming can be broken into assignment-sized chunks. This 
requires some software engineering and up-front planning by the course instructor, so that the assignment is 
well-defined in terms of the assigned code's input and output. But with careful planning, a large relatively 
large piece of software can be built gradually by students completing class assignments. 

One downside to this is that since all of the students, or student groups, are doing different assignments 
(having them all do the same assignment wastes work), the grading is more challenging. On the other hand, 
the instructor can view such grading as doing research. 

For several years in my artificial intelligence class, I gave an assignment to write a function to detect a 
spatial relationship between two objects in a photograph. Each student did a different relation (e.g., one did 
“above-below,” and another “occlusion.”) This work led to a publication (Smith et ak, 2010). 


' http://www.pbworks.com/ 
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Not all fields require programming, nor do all students have programming knowledge. However, many 
fields have some kind of data collection that can be conducted with student assignments, and all fields can 
benefit from literature reviews, which I will describe next. 

Students can be assigned to write literature reviews for topics that need them. However, writing 
literature reviews for some large topics can be too big a job for a class assignment. There are a few solutions 
to this. 

First, papers can be written by groups of students. This will make some topics manageable. 

Second, students can write first drafts of papers, at a high level of abstraction. For example, one can 
assign students to write a six-page paper that gives a very general overview of a complicated topic. This 
forces them to synthesize information and to be concise. Then, the next time the course is taught, an 
instructor can assign students to expand the paper into a 20-page paper with more detail. This new batch of 
students will have experience reviewing and re-writing other students' texts, which is also a valuable learning 
experience. 

A downside to writing programs and literature reviews is that it's sometimes difficult to know ahead of 
time whether all of the assignments are of equal difficulty. Some software ends up being very complicated, 
and others nearly trivial. An instructor might assign a literature review on a topic for which there is very little 
published. One solution is to keep in touch with the student projects as they progress, and see if the work is 
too little. If it is, the instructor should step in and expand the assignment. I require my students to give an 
in-class oral proposal of their project before too long, by which time they are usually clear about what it will 
take to complete it. After watching the presentation, the instructor can recommend that they do the original 
project, plus this or that extension, or perhaps something smaller. 


3. CONCLUSION 

My teaching philosophy is to not waste student work. To this end I have devised a number of class projects 
that contribute not only to the education of the students who do them, but for the broader educational and 
scholarly communities. My hope is that other instructors will use similar methods in their own classrooms to 
promote this kind of indirect collaboration. Before the World Wide Web, there was no mechanism for 
sharing the products of student assignments. Now that anyone can publish online, there is no excuse for 
wasting the millions of hours our students spend working on class assignments. 
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ABSTRACT 

This paper details a proposed doctoral student connections program that is based on sociocultural theory. It is designed to 
assist new students with starting their educational journey. This program is designed to leverage social interactions, peer 
mentorship, personal reflection, purposeful planning, and existing resources to assist students in navigating a 
department’s doctoral program culture. 


KEYWORDS 

Mentorship, sociocultural theory. Personal Learning Network, doctoral students, program design 


1. INTRODUCTION 

We begin this paper by proposing that doctoral students need informal as well as formal student development 
as they enter their chosen program to become effective students navigating their new learning environment 
with a minimum of challenges. Research has found that alleviating doctoral students’ challenges will shorten 
time-to-degree and thereby reduce attrition rates (West et al., 2011; Boyle Single, 2010) as well as keep cost 
down for students (Beauchamp et al, 2009). Other than cost, challenges that students often encounter include 
a) difficulty with writing (Boyle Single), b) conflicting roles as a working adult while studying (Beauchamp 
et al.), and also c) lack of social contact and advising. Mentoring, as Mullen et al. (2010: 180) noted, “can 
foster viable relationships between faculty and students, increase engagement with research and scholarship, 
[and] facilitate peer support”. Further, a well-designed doctoral program provides clear guidelines for 
students and has faculty focused on teaching and alert to student needs (Morrison et al., 201 1). Also, students 
in programs that provide clear program requirements graduate in less time (Boyle Single). We thus foresee 
that those students who understand program expectations and have ease of access to answers will be better 
equipped to maneuver their way towards successful program completion. 

Socialization is a necessary period in the doctoral years when students learn “what the academic career 
involves, the norms, values, and ethics embedded in their disciplines, and the expectations and work habits 
that they will be expected to meet” (Austin, 2009: 173). In addition to advising, faculty mentoring, and 
written requirements, peer mentoring can be a useful means to segue new students from point-to-point in 
their program. The importance of mentoring and guiding students has been documented by several 
researchers (Boyle Single, 2010; Columbaro, 2009; Kwan, 2009), however, is practiced unevenly (Sowell et 
al., 2008). The Council of Graduate Schools (CGS) has also, based on their seven-year research on issues 
surrounding Ph.D. completion and attrition, recommended that university programs a) train peer mentors to 
work with newer students, b) develop online mechanisms students can use to track their progress throughout 
the program, c) share information on the dissertation proposal process, and d) provide a FAQ page on both 
the program website and in a new student orientation handbook (Sowell et al.). 

Technologies are often described as “cognitive tools” that can transform, augment and support cognitive 
engagement among learners (McLoughlin and Oliver, 1999). Presented here is our work in progress with the 
design of the Connections Program, a doctoral development program utilizing technologies for learning, 
thinking, and connecting. It was designed to leverage sociocultural theory and include a Personal Learning 
Network (PLN) for students. It is believed that this program may help new students more smoothly transition 
between the stages of their program. 
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1.1 Theoretical Framework 

Social constructivism recognizes that knowledge is a social product constructed through discourse - a give 
and take of validity claims within cultural and historical communities - where the resulting product is what is 
agreed on as constituting truth (Prawat and Floden, 1994). Vygotsky, a noted psychologist, is recognized as 
the founder of what is known as sociocultural theory (SCT) (Holbrook, 1999). In particular, Vygotsky 
studied human consciousness as well as how socially mediated communicative acts and interactions lead to 
development of thinking and construction of meaning. SCT takes into account the cultural and historical 
aspect of learning and meaning making in a social context. As such, the theory promotes the idea that 
education should be engaged with “not just (...) theories of instruction, but with learning to learn, developing 
skills and strategies to continue to learn, with making learning experiences meaningful and relevant to the 
individual, with developing and growing as a whole person” (Williams and Burden, as cited in Turuk, 2008: 
247). In many ways, the theory challenges today’s traditional teaching methods that build on student 
regurgitation of teacher disseminated knowledge (Hausfather, 1996). 

Sociocultural theory, as noted by Holbrook (1999), includes (and is here exemplified within the doctoral 
program context): 

Social interaction within the cultural context. For example, students new to a doctoral program need 
social interactions such as language to learn and develop the necessary skills for synthesizing new 
information pertinent to their degree program. When they are forming circles of learning with faculty, with 
other new students, and with more experienced peers, new students may discuss how to fulfill the culturally 
set norms of the program. This enculturation may include: learning what it means to be a doctoral student, 
program requirements, how to conduct research, and so forth. Additionally and ideally, new students should 
strive to quickly establish a Personal Learning Network (PLN) for continued social interactions throughout 
their time in the program. The PLN is described in detail later in this paper. 

Interacting with the more knowledgeable other. In order to capitalize on the experiences and knowledge 
of more seasoned doctoral students in the program and build new knowledge blocks, new students are 
assigned peer mentors. These individuals are often referred to as more knowledgeable others (MKOs). Seen 
from this perspective, new doctoral students may benefit from this mentoring relationship because the mentor 
has been in the program longer and knows the culture and procedures of the program. Piskurich (2006) 
described the advantages and disadvantages of mentoring and noted the importance of finding mentors with 
the right skill set so that learners may truly benefit. That is to say, the interaction between mentor and mentee 
is only helpful at advancing the mentees’ cognitive skills, when the cognitive distance between the mentee 
and mentor is bridgeable. Piskurich further noted concerns and limitations with mentoring programs that 
should be taken into consideration when developing such programming. These, he said, included them being 
time-consuming, costly, difficult because of the availability of subject matter experts, and as interfering with 
other activities. 

The zone of proximal development defines the bridgeable gap between what an individual is capable of 
learning independently and what an individual is capable of learning through social interaction with MKOs 
(Vygotsky, 1978). Using a doctoral program as an example, mentors may provide scaffolding by providing 
missing information or clarifying procedures about how to submit papers for conferences or how to prepare a 
portfolio. However, if the gap is too great or miscalculated, then the interaction does not benefit either 
student. As such, the exchange must help the mentees bridge what they do not already know. 


2. PROGRAM DESIGN 
2.1 The Connections Program 

The instructional design presented here provides one possible solution for acculturating doctoral students in a 
mentoring program. The Connections Program discussed is designed to help new students form relationships 
with fellow students and faculty in their degree program. This is done in order to help novice students learn 
from their mentor’s experiences, receive help and mentorship with planning their degree completion, and to 
help them leverage social and cultural interactions to form a PLN that can continue beyond the completion of 
the Connections Program. 
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The Connections Program (Figure 1) will introduce students to the doctoral program through three main 
components: 1) a learning management organization hosted in Blackboard, 2) reflection on a blogging 
platform such as WordPress, and 3) an individualized network of learning resources and people also known 
as a Personal Learning Network. The Blackboard Organization is a container space similar to a course, 
however, not tied to only students and certain dates. This space will hold student resources and activities such 
as a mandatory Survival Basics Module for new students that pulls together into one place many of the 
disparate information sources about the program. 
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Eigure 1. Overview of the Components of the Connections. 


2.2 The Personal Learning Network (PLN) 

The PLN should afford new doctoral students the opportunity to connect with other new and existing students 
as well as with faculty. This network consists of peer mentors with program experience who also moderate 
department social media, other students with similar research interests, university staff, and any other 
resources that students utilize when learning. This mentoring network, together with the Blackboard 
Organization, should help new students discover both explicit expectations and unwritten norms regarding 
what is expected from them as new doctoral students, as well as with completing the mandatory activities of 
the Connections Program. Necessarily, the PLN requires buy-in from new students and more experienced 
students. In their role as social media moderators, peer mentors will act as guides that facilitate connections 
between new and existing students. Therefore, part of their role will include sharing information about 
newcomers, welcoming them to the community, and leveraging them into the research environment. 

When establishing this mentoring system, peer mentors will be identified by the university and/or 
academic department to be trained and rewarded for their PLN participation. Such rewards need not 
necessarily be monetary, but can potentially be something with other value. This may be the inclusion of the 
mentoring activity as an artifact in the student portfolio that showcases their personal growth in assisting 
others. In their peer mentor capacity, mentors will share their experiences with navigating the degree 
program and serve as guides within the Connections. 

New students will be expected to utilize their PLN for evaluation of the products they are required to 
complete within the program. These may include surveys, blogs, degree plan, and program success plan. The 
expectation is that implementation of these products will not increase the workload of the faculty advisor, but 
instead should naturally improve communications with students by adding additional insight into their 
progress in the doctoral program. Blogging is expected to help students start writing academically and 
professionally. This should help build important reflection and writing skills. 


2.3 Using SCT to Create the Connections Program 

The goal of the Connections Program is to provide vital resource connections that help new students 
successfully navigate their degree program to completion. The elicitation, promotion, and facilitation of 
sociocultural experiences are at the heart of the training. 
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By helping new students find answers to their various “How do I. ..and when do L...” program questions, 
peer-mentor relationships should be promoted with the expectation of capitalizing on seasoned students’ 
prior program experiences. This is expected to aid in addressing new students’ zone of proximal 
development. Additionally, this should allow a virtual program culture to be further cultivated using the 
Blackboard Organization in conjunction with social media tools (Facebook, Linkedln, Twitter, etc.). 


2.4 Learning Expectations and Assessment 


After successfully completing the Connections Program, students will have completed their Degree Plan and 
Program Success Plan as well as have formed a PLN to support their work in the program. Students will be 
expected to interact with peer mentors and faculty advisers by receiving feedback for improvement on these 
products, reflecting on the program process by writing blog entries, completing a self-paced Survival Basics 
Module, interacting with their PLN, and completing program surveys (Figure 2). They should acquire the 
skills needed to understand various aspects of their degree program, expand their role as members of the 
learning community, and utilize their PLN to aid their success in their degree program. Quantitative data will 
be collected through pre/post attitudinal surveys to measure to what degree the student is aware of the 
resources. In addition, work completed in this program will be integrated into the student’s doctoral portfolio, 
which will be used to evaluate the student’s overall success in the program once coursework is complete. 
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Figure 2. Overview of the Ten Activities of the Program 


3. CONCLUSION 

The work in progress presented in this paper is designed to alleviate some challenges that new doctoral 
students often experience at the beginning of the doctoral program. Veteran students should have many ideas 
and suggestions for new students on surviving the initial semesters of a doctoral program. Connecting new 
doctoral students with more experienced peer mentors in formal and informal settings may assist the transfer 
of knowledge from experienced students to newer ones. Additionally, creating a culture of connection and 
knowledge-sharing may improve the overall atmosphere of a program. Not every student will find this social 
model of sharing knowledge to their liking, as embracing connections may depend on individual student 
buy-in. Any department that implements a program such as this will need to adjust the process as needed 
until they get the right fit for their specific student population. While it may be argued that the social 
connections described in this paper already occur in degree programs on an informal basis, the novelty of this 
idea comes from formalizing the acculturation process to create connections with faculty and mentor 
facilitations. The integration of students’ formal learning with an informal PLN opens up the possibility of 
creating an atmosphere of learning and support from which all students may benefit. 
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ABSTRACT 

Through this paper, we describe how a doctoral program in Learning and Leadership combines the best of both worlds 
from theory based programs and applied programs. Participants work from their embedded professional practice 
underpinned with the theoretical constructs of the program's seven foundational competency domains. 

Competencies are characterized as behaviors, knowledge, skills and abilities, exhibited in professional practice, and can 
be objectively measured, enhanced, and improved through learning, both formal and informal. Demonstration of 
competency (or lack thereof) directly impacts the success of individuals and organizations. Core competencies, for 
purposes of our program, are areas of focus upon which future learning and professional practice will be developed. 
Using core competencies as the baseline for a degree program enhances knowledge acquisition and performance. 
Competencies should not be static, but are continually enhanced through academic engagement, experiential learning and 
ongoing professional development, and serve as progress indicators for the evaluation and assessment process. 

The Learning and Leadership Doctoral Program is grounded in the following seven competency areas. Upon completion 
of the coursework, participants are expected to exhibit demonstrable competence in each area, as well as the ability to 
synthesize the connections between the seven domains. 

• Learning • Organizational Effectiveness 

• Leadership • Technology and Innovation 

• Research • Communication 

• Measurement 

Using a process of weaving experiential learning with theoretical constructs throughout the program’s core coursework, 
participants demonstrate their competence in the program domains in an ongoing manner that culminates with the 
Comprehensive Assessment; a presentation of a Digital Portfolio cataloguing their demonstration of competence through 
Critical Reflections on each of the program domains including artifacts to support the demonstration of competence and 
related experiential learning. This Comprehensive Assessment is held with a team of at least three faculty members and 
includes both written and oral components. 

The session concludes by reviewing lessons learned, best practices, and opportunities for further program development. 


KEYWORDS 

Experiential Learning, Demonstrable Competence, Competency Domains 


1. INTRODUCTION 

In our Doctoral Program in Learning and Leadership, the Comprehensive Assessment represents the 
blueprint for evaluation of progress in and completion of the program. The Comprehensive Assessment is a 
work-in-progress evidenced by a Digital Portfolio of specific artifacts used to demonstrate competence in 
seven program domains, including a detailed Critical Synthesis Paper (CSP). The oral presentation of the 
Digital Portfolio (written components) serves as the Comprehensive Assessment in the program, prior to 
moving to the dissertation stage. Consequently, it is imperative that each participant’s Digital Portfolio 
demonstrates evidence of competency weaving theoretical understanding and fluency with knowledge of and 
reflection on seminal works associated with the competency, specific experiential learning, best practices, 
and practical application in each of the competency areas. The Digital Portfolio will evolve in quality and 
quantity during the program. Throughout the doctoral program, each participant reviews, along with the 
program advisor and the core faculty, the progress being made and revises and updates the plan as needed. 
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2. DEMONSTRABLE COMPETENCE 

This paper describes how a doctoral program combines the best of both theory based programs and applied 
programs and how competence in the program domains is demonstrated through core coursework, reflective 
practice, and the Comprehensive Assessment. Participants work from their embedded professional practice 
underpinned with the theoretical constructs of the program's seven foundational competency domains. 

2.1 The Competency Domains 

Competencies can be characterized as behaviors, knowledge, skills and abilities, exhibited in professional 
practice, and can be objectively measured, enhanced, and improved through learning experiences, both 
formal and informal. Demonstration of competency (or lack thereof) directly impacts the success of 
individuals and organizations. Core competencies, for purposes of our program, are areas of focus upon 
which future learning and professional practice will be developed. Using core competencies as the baseline 
for a degree program enhances knowledge acquisition and performance improvement. Competencies can 
serve as progress indicators for the evaluation and assessment process. Competencies should not be static or 
fixed, but are continually enhanced through academic engagement, experiential learning and ongoing 
professional development, and can serve as progress indicators for the evaluation and assessment process. 

The Learning and Leadership Doctoral Program is grounded in seven competency areas. Upon 
completion of coursework, participants are expected to exhibit demonstrable competence in each area, as 
well as the ability to synthesize the connections between the seven domains. 

• Learning • Organizational effectiveness 

• Leadership • Technology & innovation 

• Research • Communication 

• Measurement 

2.2 Demonstrable Competence 

Our definition of competence is the knowledge that enables a person to communicate or demonstrate a given 
concept or construct. Demonstrable Competence is articulating that knowledge through thoughts, actions and 
behaviors to indicate thorough understanding and abilities in a particular subject matter construct. 

For example, if a student were studying quantitative subject matter, s/he would be expected to have the 
ability to select the appropriate quantitative methods, perform calculations, and then use the resulting data to 
perform analyses. Thus one is demonstrating competence in that subject matter by being able to do the 
calculations (skill demonstration) as well as to provide the analysis, which goes beyond the demonstration of 
skill and into competence. Competence is not only knowing how to do something; but it is also understanding 
what to do and why. This includes knowing why, how, and what behavior you need to engage in as a result. 

Demonstrable competence is the logical way to measure evidence of learning. We view evidence of 
learning as showing a behavioral transformation as a result of the experience. Learning is synonymous with 
transformation. We believe demonstrable competence is the critical element to support stakeholder 
acceptance of competency-based learning versus content-based learning in graduate degree programs. 

2.3 Experiential Learning Eoundations 

The foundation for demonstration of competence is based in the program’s underlying belief that experiential 
learning is the basis for weaving theory with professional practice. Experiential-learning theories gained in 
strength and influence in the arena of educational thought starting in the late 1900’s (Bond et al., 1985; Bond 
& Walker, 1991; Freire, 1970; Jarvis, 1987; Knowles, 1980; Kolb, 1984; Merriam & Brockett, 1997). 

David Kolb (1984), in his seminal treatise on experiential learning, defined learning as “a process 
whereby knowledge is created through the transformation of experience” (p. 38). Kolb described this process 
of adaptation and transformation through a learning model in a four-stage cycle. The first stage involves the 
learner engaging in concrete experiences. The second stage allows the learner to use the experience as a basis 
for reflective observation. 
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From this reflection, the learner then moves to Kolb’s third stage of building an idea, a generalization, or 
theory about the experience and reflection. The fourth stage, active experimentation, comes as the learner 
tests the application of the new theories in order to solve practical problems. The cycle also depicts four 
different kinds of learning environments, which in turn demonstrate the learning styles or preferences of the 
student at each stage of learning. Kolb claimed, however, that it is from the utilization of all four stages of the 
cycle that meaningful learning occurs (Kolb, 1984; Merriam & Brocket!, 1997). 

2.4 Assessment of Experiential Learning 

Palomba and Banta (1999) defined assessment as “the systematic collection, review, and use of information 
about educational programs undertaken for the purpose of improving student learning and development” (p. 
4). They advocated a view of assessment that “provide[s] students an opportunity to learn from the aetivities 
in whieh they engage” (p. 336). Case (1992) categorized authentic assessment into three types ineluding 
performance assessment, naturalistic assessment, and portfolio assessment. There are a number of other 
definitions for these types of assessment. Portfolio assessment has come to be widely used in adult learning 
as a method that can also include performance and naturalistic assessment (Barnett & Lee, 1994). Authentic 
and performance assessment have often been called alternative assessment. All of the terms have two 
characteristics in common - first, they are all viewed as alternatives to traditional multiple-choice, 
standardized achievement tests, and second, they all rely on direct examination of performance on relevant, 
real-life tasks (Worthen, 1993). That alternative or authentic assessment emerged in the 1990s as a valuable 
approach to educational evaluation is evidenced by the body of research on the subject (Bol, Stephenson, 
O’Connell, & Nunnery, 1998; Madaus & Kellaghan, 1993; Maeroff, 1991; Meyer, 1992; Worthen, 1993). 

2.5 Portfolio Assessment 

Many studies describing portfolio usage in graduate education are connected to teacher education (K. Wolf, 
1991). Mills and Reisetter (1995) described their experiences with authentic assessment through the use of 
portfolios in graduate classes in the field of education. They discussed failures and successes and the need to 
achieve a delicate balance between educating graduate students in what materials should be included in the 
portfolio, when to prescribe materials to be included, and how to encourage students to be creative in their 
own selection of materials. Macisaac and Jackson (1994) also discussed the idea of balance between 
prescribed material and the free choice of inclusions. They advised that while learners should be permitted 
flexibility in constructing portfolios, the portfolio process and its products must be configured in a manner 
that enhances the way information is communicated to portfolio users. Further, the authors explained that 
structured or required materials make it easier for adult learners to substantiate the learning that has taken 
place, showing competencies gained in cumulative learning and demonstrating personal growth and change. 

2.6 Using Demonstrable Competence to Measure Learning 

From the outset of the doctoral program, participants are encouraged to begin the demonstration of 
competence. This is accomplished through a number of different types of course deliverables, each designed 
to allow the participant to weave his/her experiential learning with the concepts and constructs discovered 
and discussed in the various aspects of coursework. 

Discussion Forums - The program utilizes a hybrid based delivery system that includes extensive use of 
the Virtual Classroom (Rausch & Crawford, 2012). After a preparatory broad spectrum of seminal and 
scholarly theoretical readings, initial face-to-face sessions are facilitated where theory is reinforced and 
foundations for learning established. Unlike traditional content based learning where a question based on the 
readings is thrown to the class from the podium, and responses can rarely be reflected upon thoughtfully and 
reasoned as no time is allowed for processing, imagine posing a question in a discussion forum and asking 
the students to pick up the discussion over the next 24 hours as they reflect upon their own life experiences 
and how the theoretical constructs introduced in the class may impact their current view of those experiences. 

Critical Reflections - In each core course (based on seven competency domains), participants are asked to 
write a Critical Reflection paper. The purpose of the paper is to demonstrate competence and ultimately 
mastery of a specific program competency area including the specific learning about theories and concepts. 


400 



lADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2013) 


These Critical Reflection papers serve as "cover documents" representing each competency area, and are 
used to weave the theoretical understanding and fluency together with knowledge of and reflection on the 
seminal works literature associated with the specific competency, and the participant's specific experiential 
learning, accompanied by practical application in each specific competency area. The Critical Reflections are 
set with a minimum of 1500 words and typically include three entwined elements: a) a summary of the 
learning experience - including what the participant has done related the competency area and what the 
measureable outcomes of the experience were; b) identification and explanation of the relevant theoretical 
constructs and how they relate to the competency domain - including a demonstration of “ownership” of the 
identified theory; and c) utilization of the concepts and theories to explain the learning experience - including 
demonstrating the ability to successfully and insightfully apply the theory to practical application. 

Competency Plan - While the foundational competencies of the doctoral program are Learning and 
Leadership, all seven competencies are important. One way that we work with participants to better 
understand and articulate the integrated nature of the competencies is to encourage them to separate the seven 
competencies and view them in isolation through the Competency Plan and then put them together in a final 
synthesis as their program unfolds. The Competency Plan is organized around the following questions: 

• What is my background/history relative to this competency area? Experience in the area, courses 
related to the area, and reflections on my current competence in this area? 

• What level of competence do 1 want to achieve in this competency area? Published, scholarly, and 
leading-edge competence? Professional-practice competence? 

• What is my plan to achieve the desired level of competence in this area? Academic plans, 
professional plans, collaborative plans, personal plans? 

• How do 1 intend to integrate each of the program’s required courses into my overall plan? 

Digital Portfolio - Demonstration of competence is documented via a Digital Portfolio that the participant 
designs, creates and constructs throughout the program. The purpose of the Digital Portfolio is to document 
and assist in assessment and evaluation of the participant's level of personal growth and subject matter 
competence as reflected in each of the seven competency areas represented in the program, including artifacts 
and materials that demonstrate how the participant has developed competence and ultimately mastery 
through application in professional practice. The Digital Portfolio is a tool which assists the participant in the 
collection and organization of artifacts and materials that reflect his/her progress and achievements during the 
doctoral program. The specific method of how the participant collects and organizes the Digital Portfolio is a 
matter of his or her choice; however, most choose to organize by competency area. Each section of the 
Digital Portfolio includes artifacts and materials that demonstrate how the participant has developed and is 
developing competency in the specified domain. The Competency Plan provides an outline of the materials to 
be collected and organized in the Digital Portfolio. The Digital Portfolio is a practical record that chronicles 
the participant’s growth in Learning and Leadership. The participant works closely with program faculty and 
advisor to determine artifacts that are relevant to the Digital Portfolio documentation. Items included in the 
Digital Portfolio should be carefully selected and should tie directly to the competency domains. 

The Critical Synthesis Paper (CSP) is the culminating manuscript demonstrating the participant’s 
knowledge and understanding and a complete synthesis of all seven competency areas. Its purpose is to 
exhibit knowledge of the academic discipline associated with the competencies and confirm effective 
analytical abilities and writing proficiency in a holistic fashion, not just a paper that bolts seven competencies 
together or is merely a report on each competency area. The CSP reflects the participant’s personal journey in 
the Learning and Leadership program. 

The Critical Synthesis Paper should be 4000 words (minimum) in length. The Critical Synthesis Paper 
(CSP) contains the following elements: 

a. A personal reflection on the subject of learning and leadership: What your definition of learning and 
leadership was in the beginning, what it is now, and how it has changed. 

b. Relative to “learning” and “leadership,” who you were as you began the program at Induction, how 
you have changed, and who you are now. 

The Critical Synthesis paper is where the participant weaves together his/her understanding and 
demonstration of competence across the seven program domains, with an emphasis on the foundational 
domains of Learning and Leadership. The participant’s ability to intertwine and make experiential 
connections among the seven domains is the ultimate step in the demonstration of competency. 
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3. CONCLUSION 

Over the past 7 years since the program begin, we have learned a number of lessons relative to adopting 
Demonstrable Competence as the method for assessment of learning in the program. One technique that 
seems to be particularly successful, as well as appreciated by the participants in the program, has been 
building the process of demonstrating competence from the outset of the program into each core course. By 
using the “rough draft” of the Critical Reflection for the program domain in the initial core topic courses, 
participants begin to weave their experiential learning with the theoretical constructs of the coursework from 
the very beginning. This serves in two ways. First, they are able to begin building the framework for the 
Comprehensive Assessment early, and by the time they have completed the core coursework, the structure of 
the written parts of this culminating deliverable is largely in place. Secondly, they are learning the process of 
demonstrating competence in the program domains in the earliest stages of the program and this process 
builds skill, competence, and confidence, allowing and even encouraging continuous success. 

Program experiences reinforce that Demonstrable Competence is truly an expression of learning through 
thoughts, actions and behaviors that represent thorough understanding and application in a subject matter 
domain. We view evidence of learning (demonstration of competence) as being able to show a behavioral 
transformation as a result of the experience. We believe demonstrable competence is the critical element for 
competency-based versus traditional content -based learning in graduate programs. 
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ABSTRACT 

Traditional knowledge assessments rely on multiple-choice type questions that only report a right or wrong answer. The 
reliance within the education system on this technique infers that a student who provides a correct answer purely through 
guesswork possesses knowledge equivalent to a student who actually knows the correct answer. A more complete 
assessment is needed to eliminate guesswork and offer greater efficiency in managing the remediation process. 
Confidence-based assessment techniques integrate the selection of multiple-choice answers with the student’s 
self-perceived level of certainty and offer a middle ground between the traditional multiple-choice answer and a lengthy 
essay response. Research has discovered that confidence-based assessments provide a more comprehensive measure of a 
person’s knowledge, increases the retainability of learned material and identifies topics in which people are misinformed. 


KEYWORDS 

Confidence-based, assessments, certainty-based, exams 


1. INTRODUCTION 

The traditional use of multiple-choice questions reward a student for guessing. Students are often told when 
preparing for an exam that even if they are unsure of the correct answer, they should answer it anyway, 
because with a multiple -choice selection there is a 20% (5-choice) to 25% (4-choice) chance of guessing the 
correct answer. Their odds are even better when guessing on a true/false choice; hence the reason students’ 
poke fun at the process, calling it a “multiple-guess” exam. There is an effort to maximize the score instead 
of gaining an understanding of the course material. But in this world of number-crunching rationalizations, it 
requires much less effort to assign a number (numeric test score) to represent a level of understanding that 
can be quantified, studied and managed. Is this really an effective way of measuring a student’s 
comprehension of a complex subject? 

Some research has been performed with explorations into the implementation of confidence -based 
assessments across various academic disciplines. Most notably, the work of Darwin Hunt, James Bruno and 
Tony Gardner-Medwin has provided exhaustive research in the field, with many of the assessment techniques 
moving into more focused research and commercial operations. Dr. Darwin Hunt began his research in the 
early 1980s, discovering highly correlative measures between human self-assessment and learning. His work 
since then has continued to provide evidence that a confidence-based approach to knowledge assessment 
provides a more comprehensive measure of a student’s knowledge, including the retainability of learned 
material. 

Hunt (2003) found evidence of the obvious, that when a student is given a selection of answers for a 
simple arithmetic problem, he or she still has a chance to select the correct answer even if they don’t know 
how to add two numbers. The regrettable part is that the reliance within the education system on this 
technique infers that a student who provides a correct answer purely through guesswork possesses knowledge 
equivalent to a student who actually knows the correct answer. This situation is even more damaging (Adams 
& Ewen, 2009), as it presents numerous problems for academic institutions in their attempt to offer a fair and 
representative evaluation of a student’s knowledge that can be compared against a standard. 

Guessing on a few questions in Math-101 may be perceived as somewhat benign, but taken to its eventual 
limits, the most damaging aspect of this guesswork is within a safety-critical environment. 


403 



ISBN: 978-989-8533-18-0 © 2013 lADIS 


Many industrial accidents, injuries and even deaths have been caused by misinformed operators who held 
a steadfast belief that a factoid of knowledge was correct when it was indeed wrong. 


2. CONFIDENCE-BASED ASSESSMENTS 

Traditional knowledge assessment methods attempt to focus on the recall of previously-presented 
information. Often these assessments rely on multiple -choice exams, tests or quizzes that only measure a 
right or wrong answer. But exploring the real assessment of knowledge needs to eliminate any guesswork 
involved. This issue is generally mitigated with a short list of techniques that are often ineffective or 
burdensome. One of the more popular techniques is to give the student a very long test, with upwards of 50, 
or more, multiple-choice questions. This method is somewhat effective in reducing the negative aspect of 
missing a few questions, and also offers a more uniform assessment of knowledge because multiple questions 
can be asked about a single topic from different directions, but the long test takes a lot of time to complete 
and is quite overwhelming to apprehensive students. Another method of mitigating guesswork is to 
administer questions that require fill-in, or essay type answers, but these techniques are again daunting to 
students and burdensome to grade with large class sizes, not to mention the problems associated with a 
subjective evaluation grading scheme. 

Some studies have identified the lack of knowledge retention even after a student successfully passes 
traditional assessments. Adams et al. (2009) postulated that, “...even when students pass these [traditional] 
assessments, they lack the necessary skill set to perform well in the work place (p. 1).” This infers that even a 
high test score is not a sufficient indicator of subsequent job performance, although as a society, we generally 
hold this view. Many variables are involved when attempting to predict job performance, with knowledge 
retention being a major contributor. 

Adams continues, “For years we have been teaching and assessing using traditional models that 
encourage guesswork in the testing process. There are even strategies on test taking provided to students to 
maximize their score, rather than gain an understanding of their knowledge acquisition, skills and 
competency to perform (p. 1).” An entire cottage industry has evolved around the “just get me through the 
test” cram course. 

A common practice among entrance exam or certification test takers is to purchase test-prep books or 
software that are just reprints of test question banks. This rather defeats the purpose of an exam in the first 
place, since answers can be easily memorized. It does not promote even the slightest hint of understanding, 
much less correlation of the course material. The question beckons, do we really want question memorizers 
to operate the machinery of our industrialized society? 

2.1 Confidence-Based Assessment Method 

Short of turning every exam into a drawn-out creative writing exercise, the implementation of a confidence- 
based testing methodology is a way to effeetively gain a quality assessment of a student’s knowledge 
retention while still being able to quantify the results. Confidence-based assessments offer a middle ground 
between the traditional multiple-choice answer and a lengthy essay response. While taking a multiple-choice 
test, students indicate which answers they believe are correct while also indicating how confident they feel 
with their selections. A combined composite score is the result, with a rating scale technique used to reduce 
the variables. 

The combined score assigns greater weight to more confident right answers, and penalizes highly- 
confident wrong answers. This scheme effectively eliminates overconfident learners who assign a high 
confidence level to all their answers. Upon analysis by the instructor, or self-realized by the student, a 
confident wrong answer would deserve special attention and greater emphasis during follow-up learning 
sessions. The effect is to eliminate, or severely reduce, any guesswork to reveal a true assessment of a 
student’s knowledge. 

2.1.1 Marking System 

The instrument of a confidence-based assessment scheme uses a double-tiered marking system to gain two 
distinct but interrelated scores in response to a specific question, although its effectiveness is dependent on 
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the type of knowledge being assessed. Within a testing scheme of clearly black and white options, such as 
math, hard science or medical subjects, the confidence-based assessment model provides easily-quantifiable 
results. Whereas, in the soft sciences or art related subjects, the confidence-based assessment model may only 
lend support to traditional (subjective) evaluations. 

Once a test taker marks the answer which they believe is correct, they are asked to mark their level of 
confidence. Previous studies have used either a three-level choice (Gardner-Medwin & Curtin, 2003a & 
Bruno, 1995) or a five-level choice (Hunt, 2003), providing ease of use and sufficient data spread for 
statistical analysis. Both scales have been used in experimental trials, but in a study that specifically 
monitored the range of certainty markings, Gvozdenko et al. (2007) discovered that 85% of his student 
subjects preferred the shorter scales. Additional studies still under investigation are experimenting with 
variations on this methodology. 

2.1.2 Grading Scheme 

Table 1 shows the grading scale used by Gardner-Medwin & Gahan (2003b) within an active biomedical 
student curriculum. The cumulative grade using the certainty-based grading scale assigns a grade straight 
across for low certainty, but in the medium and high certainty levels, bonus points are awarded for correct 
answers, while a greater proportion of points are deducted for wrong answers. 

Table 1. Grading scales used by Gardner-Medwin & Gahan (2003b) in a biomedical student curriculum 


Degree of 
Certainty 

C=1 

Low certainty 

C=2 

Medium certainty 

C=3 

High certainty 

No reply 

Mark if 
Correct 

1 

2 

3 

0 

Penalty if 
wrong 

0 

-2 

-6 

0 


Gardner-Medwin successfully employed this three-level confidence scale, where the upper two certainty 
levels apply negative weights for wrong answers, effectively assigning a greater penalty for each higher level 
of misinformation. This gradation is critical, and weighs heavily in the test taker’s decision for each answer, 
offering motivation for a more reliable self-assessment. 

An important element within this confidence-based assessment scheme is that test takers know and 
understand the marking methodology, especially that points will be taken away from their final score for 
highly-confident wrong answers. If the confidence-based scheme is going to be used as a formative 
assessment, then the test taker must be informed of its use as a self-assessment tool and how it could be used 
effectively in their own motivation and appraisal of future learning parameters. 

2.1.3 Benefits 

From the students’ perspective, when offered the use of an interactive assessment that integrated confidence 
awareness, the instant feedback allows them to quickly confirm their beliefs in selecting the right answer or 
to correct their perception of wrong answers. This scheme offers a learning tool that would greatly enhance 
their knowledge absorption and retention. (Gardner-Medwin et al. 2003a, 2003b & 2006) 

Likewise, the teacher can obtain a grade report across an entire class, which can then be used to determine 
the teaching effectiveness of the courseware materials or teaching techniques across groups. Although that 
effectiveness is dependent on close control of group variables, it can easily determine that all-important 
check of a student’s, or group, knowledge level either at the beginning of a course, or at pivotal points within 
the course. The information can then be used to modify the course curriculum or place special emphasis on 
weak areas. 

The eventual goal is that the individual student completes the course with a higher retention rate and is 
fully prepared for the next course in the series. Adams et al. (2009) concluded that, “...confidence and 
knowledge are correlated and are both critical determinants in evaluating future performance (p. 2).” Hunt 
(2003) also supports that self-assessment testing provides measurements more closely related to a person’s 
later performance than the common multiple-choice test. But, there are limitations to the process, as Khatibi 
et al. (2010) found gender differences in a retrospective study of biochemistry students’ summative exam 
scores. The female students’ confidence -level scores were significantly higher than the males, suspecting 
different psychological characteristics as a factor. 
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Yen et al, (2010) compared the addition of a confidence-weighted component to a computer- 
administered, multiple -choice exam, finding a correlation between confidence and individual abilities. 
Additionally, their study found that confidence-based exams became overall more efficient, requiring fewer 
individual questions to estimate examinees’ ability. 

The use of a confidence-based assessment scheme appears to be very popular with students, especially 
when offered as a formative tool during a course, as they see it as a method to improve their overall grade. 
Gvozdenko et al. (2007) quantified this when monitoring the use of the certainty scale in formative and 
summative testing. The study found that, “...the majority of students (67%) on a summative test and nearly 
all students (96%) on a formative test chose to use the certainty scale as an additional tool in the testing 
procedure (p. 215).” 


3. CONCLUSION 

It is safe to conclude that the implementation of a confidence-based assessment scheme is beneficial in 
achieving the objectives of a course when used as a formative evaluation tool. Confidence -based assessments 
offer a middle ground between the traditional multiple-choice answer and a lengthy essay response, resulting 
in a quality measure of a student’s knowledge retention while still being able to quantify the results against a 
standard. Additionally, the confidence -level marking scale is easily understood and accepted by students 
when it was used in secondary and post-secondary schools. 

Much of the functionality of a confidence-based assessment platform is already integrated into popular 
Learning Management Systems (LMS). A few well-established academic LMS platforms (Blackboard, 
Moodle, Schoology, etc.) contain assessment modules, but Moodle (www.Moodle.org) has already 
implemented a robust confidence-based assessment module, offering a wide variety of development options, 
which has been used by Gardner-Medwin in both his formal research studies. 

The researchers throughout this review primarily explored the effects of a confidence -based assessment 
scheme within secondary and post-secondary school environments. Little research has been performed on the 
effects within a purely adult-oriented, corporate or skills-based technical learning environment. Further 
studies outside of academia may uncover additional benefits or limitations associated with these non- 
traditional situations. Additionally, it is suspected that there may also be cultural implications when 
implementing a confidence -based assessment tool, especially within Asian cultures who generally find it 
unnatural to rate themselves above the average. 
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ABSTRACT 

Results of a collaborative, quasi-experimental, research and development project partnering university professors with a 
seventh grade science teacher are reported. The study proposed to test the academic effectiveness of innovative 
digitally-inspired instruction using commonly available digital tools on 33 North Texas public school students enrolled in 
seventh grade science. Technology was introduced into science lessons not as the goal of instruction, but as tools to 
develop effective and engaging 2P‘ Century learning. State mandated student learning objectives were tested pre- and 
post-intervention using district benchmark tests to determine effects of digitally-inspired instruction on achievement. 
Mean achievement pass rates increased from 18% - 42% for all learning objectives measured. Instructional unit lesson 
design and associated digital tools are also described. 


KEYWORDS 

Science Achievement, Digital tools, BYOT 


1. INTRODUCTION 

Contemporary K-12 students adept at multitasking in fast-paced, multidimensional digital environments often 
disengage from traditional two-dimensional instruction when technology merely substitutes for paper/pencil 
tasks (i.e. type an essay) or occasionally augments 20* Century instruction (i.e. view a YouTube video; 
Puentedura, 2008). To re-engage 2P‘ Century learners, digital instruction models that evolve from current 
technology are needed (Taylor, 2005). Educators inspired to experiment with engaging new instructional 
models need support from school administration and university researchers to develop and test the 
effectiveness of instruction not previously experienced or observed (Hayes, 2010; November, 2010). 


2. REVIEW OF LITERATURE 

Literature supports the need for improvement in science instruction in the US and in Texas (National Center 
for Education Statistics, 2012; TIMSS, 2007). Science, technology, engineering, and math (STEM) research 
recommends development of contemporary, research-based teaching strategies and materials to support 
active, in-depth, global learning (Dickman, Schwabe, Schmidt, & Henken, 2009; Lopatto, 2007; National 
Research Council, 2005; 2007; 2009; Wood, 2009). Eurthermore, STEM curriculum should challenge 
students with open-ended assignments that are both personally meaningful and engaging (Barak & Asad, 
2012). 

To evolve from passive content-consumers to active information-processors requires instructional 
engagement. Engaged learners work collaboratively, transforming understanding through creative problem 
solving (Jones, Valdez, Nowakowski, & Rasmussen, 1994). Wasserstein (1995) noted authentic engagement 
occurs when educators furnish students with enough skills and tools to become self-motivated. Schlechty 
(2001) stresses students learn best in applied learning tasks, emphasizing engagement is an active and 
interactive process, and not synonymous with time on task. Engaged students learn more, retain more, and 
enjoy the learning activities more than unengaged students (Dowson & Mclnerney, 2001; Hancock & Betts, 
2002; Lumsden, 1994; Yoke, 2002). 
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Instructional goals that create opportunities for authentic engagement, where students meet expectations 
and intended instructional outcomes responsive to learner interests and values, produce the most effective 
learning (Schlechty, 2002). 

Although technologies have tremendous potential to transform learning experiences, empirical evidence 
does not support instructional effectiveness when technology merely augments content delivery (Cuban, 
2002; Cuban, Kirkpatrick, & Peck, 2001; Judson, 2006; McClure, Jukes, & MacLean, 2012; Palak & Walls, 
2009; Windschitl & Sahl, 2002). Conversely, use of digital tools coordinated with effective 
research-supported instructional practices can promote collaborative learning environments focused on 
student engagement and in-depth conceptual investigation (Freidman, Beauchamp, Blain, Lirette-Pitre, & 
Fournier 2011; Kahveci, 2010; Keser, Uzunboylu, & Ozdamli, 2011; Smeureanu & Isaila, 201 1). 

It is evident more research is needed to understand how to design technology-infused, learner-centered 
instruction. The purpose of this study was to determine the effect of implementing transformational 
digitally-inspired instruction on public school science students’ science achievement. 


3. METHOD 

Applying a quasi-experimental research and development model, university researchers partnered with 
public school teachers to develop and test lessons that incorporated digitally-inspired instruction using 
devices common in a student’s environment (smart boards, computers, tablets, smart phones, etc.). The 
sample consisted of 33 students enrolled in one teacher’s seventh grade science classes at a North Texas 
middle school. In addition, 19 fifth grade, and 29 fourth grade students from three classes in other schools 
participated in the last lesson of the unit and served as an authentic audience for the culminating project. 

Based on state mandated learning objectives, a four-week instructional unit was developed and 
implemented. Learning objectives included the main function of the systems of the human organism, levels 
of organization in plants and animals, difference between the structure and function in plant and animal cell, 
functions of the cell and cell theory of all organisms. Students were pre- and post-tested on benchmark 
questions from the standardized Texas seventh grade assessment based on learning objectives for seventh 
grade science to determine the effect of unit instruction on achievement. The classroom teacher coordinated 
the collection of student academic achievement data following the district procedures for benchmark testing. 
Pre- and post-test scores were submitted to university researchers for analysis using online data collection 
forms (Google Docs) at the conclusion of the instructional unit. 

The four-week unit implemented, entitled “The Organ Trail,” included the following activities and 
associated digital tools. In weeks 1 and 2, Searching multiple digital information resources, students 
completed webquests to answer guiding questions related to the purpose and importance of one organ of the 
human body (as assigned by the teacher). Next, in the role of employees working in teams, groups of 2-3 
students collaborated to create a letter addressed to the Human Body Corporation using evidence to justify 
why their assigned organ should not be fired from the Human Body. To compose the letter, students used 
Meetingwords (http://www.meetingwords.com) which works like a wiki but each contributor is color-coded 
and there is a built in, back channel for teacher monitoring. After completing the letter, students used the 
Baiboard (http://www.baiboard.com) application to create a collaborative poster to graphically support the 
arguments in their group letter. 

In weeks 3 and 4: Learning activities culminated in two frog dissection exercises. In the role of surgeons, 
students completed a self-paced digital frog dissection program using the Digital Frog 2.5 APP 
(http://www.digitalfrog.com) and a Frog Dissection APP purchased within the Edmodo platform (like the 
iPad APP, but not limited to one purchase). In addition to completing the associated digital workbook 
activities, the teacher embedded questions requiring students to describe what they learned about their own 
organs by studying the frog's organs. After completing the digital frog dissection activities, students dissected 
real frogs. Three elementary science classes (two fourth and one fifth grade) from different schools viewed 
the dissections using SKYPE projector mode with one webcam. Skyping was included to replicate the 
experience of global collaboration. Sharing iPads, pairs of elementary observers used a Todaysmeet 
(http://todaysmeeet.com) back channel session to ask questions. One seventh grade student mediated the 
Todaysmeet feedback and relayed fourth and fifth grade student questions to seventh grade dissectors who 
responded directly. 
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4. RESULTS 

Students (N=33) were pre- and post-tested using benchmark questions from Texas learning objectives for 
seventh grade science to determine the effect of unit instruction on achievement. The benchmark passing goal 
for each learning objective was set at a score of 70%. Findings indicated an increase in the percentage of 
students who reached the 70% achievement level from the pretest to the post test on all objectives tested. 
Nine questions regarding the main function of the systems of the human organism saw the number of 
students who met the 70% pass rate increase 18% from pretest (64% passed) to posttest (82% passed). On 
four questions pertaining to levels of organization in plants and animals, the number of students meeting the 
70% pass rate increased 33% from pretest (67% passed) to posttest (100% passed). Differences between the 
structure and function in plant and animal cells, addressed in six questions, increased student pass rates 42% 
from pretest (52 % passed) to posttest (94% passed). Three questions pertaining to the functions of the cell 
saw a 34% increase in students who passed from pretest (58% passed) to posttest (82% passed). Finally, on 
two questions related to cell theory of all organisms pass rates for students increased 21% from pretest (64% 
passed) to posttest (85% passed). 


5. CONCLUSION 

Improved academic achievement can result when technology is strategically used to meet instructional goals. 
Through interaction and exploration in creative and innovative ways, technology empowers students to 
communicate and socialize beyond the classroom. As evidenced by the results of this study, a digitally 
expanded classroom can accommodate community-driven, interdisciplinary, and virtual collaboration. Digital 
tools provide an unprecedented opportunity for schools to reexamine traditional approaches and current 
practices, and redesign parameters of effective instruction (The Horizon Report, 2012). Supporting educator 
effectiveness by expanding innovative learning models that utilize online and blended learning, high-access, 
technology-enriched learning environments, and personalized learning models will likely increase student 
learning (State Educational Technology Directors Association, 201 1). Furthermore, technology used to 
redesign or create original tasks (rather than replicate tradition instruction) results in richer, more engaged 
and integrated learning at higher levels of thinking (Puentedura, 1980). 

Teaching digital learners demands different instructional strategies. Educators today must engage digital 
learners and create instructional opportunities by utilizing technology to empower learners. Schools must 
move from automating processes (attendance, report cards, email) to informating processes that empower 
students to solve problems, access information and create relationships outside the classroom using the tools 
of technology (November, 2010). 
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ABSTRACT 

Two alternative technologies forming the basis of computer-mediated teacher preparation systems are compared and 
contrasted regarding implementation, operation, and assessment considerations. The role-playing system in Second 
Life is shown to have the unique characteristic of developing a co-constructed pedagogical identity, while the flight 
simulator metaphor of simSchool encourages rapid, stepwise refinement of pedagogical expertise. Each has cost and 
traveling distance advantages over face-to face traditional meetings, as well as some shortcomings. Ultimately, the 
largest assessment issue for both is how to measure learning inside a simulator or a social media space. Eurther research 
is needed in this area. 
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1. INTRODUCTION 

In spite of efforts to better prepare teachers and support their induction into the profession, the high attrition 
rate of teachers in their first five years suggests persistent problems. Bright young teachers are leaving at an 
unsustainable rate (National Commission on Teaching and America's Future, 2007). Many of these problems 
concern self-efficacy in terms of classroom management skills (Greiner, 2009). This article introduces two 
highly interactive technologies aimed at giving pre-service and new teachers improved classroom 
management preparation. Both technologies involve simulations that address the challenge of providing 
pre-service teachers with experiences interacting with the wide variety of student behaviors they will 
encounter in the real world of teaching. One approach involves a Second Life environment in which 
pre-service teachers play either the role of a classroom teacher or a student in a classroom. Afterwards, the 
class de-briefs and discusses the behaviors and teacher responses, and makes suggestions for alternative 
actions. The second approach involves simSchool, a flight simulator for teaching that uses a computational 
model of teaching and learning. simSchool supports practice and reflection on a variety of teaching 
challenges, including classroom management, classroom activity design, and the psychology of learning. 
Each technology - role playing and computational modeling - will be introduced in its own section, followed 
by comparisons and contrasts between the two. The discussion section will first summarize implementation 
considerations and then focus on assessment issues arising when comparing the two. 
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2. CONCEPTUAL RATIONALE 

Digital simulations offer a promising new way to provide a practice environment for student teachers 
(Grossman, 2010); they provide low-risk, high-touch, scalable and efficient method for microteaching and 
pedagogical experimentation by integrating the elements of fantasy and play with realistic dynamics and 
authentic actions into the pre-service classroom. The two methods presented here, a virtual world for 
role-playing and a computational model of learning embedded in a game-like interface, each have different 
affordances for virtual professional practice, introduced below. 

Among the new affordances of simulation-based professional practice, new assessment opportunities 
stand out. As seen in the pilot study of the virtual role-playing environment, students negotiate their 
pedagogical identities revealing some evidence that while microteaching and observing other students 
experimentations, they construct, deconstruct and reconstruct their conceptions of the dynamics at play in the 
classroom. Of interest is to what degree the simulations and co-constructed understanding of those translate 
into the actions of an efficacious beginning teacher. Further study, perhaps following a group of students 
from early fieldwork, through student teaching and into the classroom might reveal positive effects of virtual 
play combined with co-construction of identity work. 

In the case of interacting with simSchool, the assessment challenge includes how to make sense of what a 
user knows and can do based on an analysis of interaction log files, providing a view of the user’s 
performance over time. Today, a person responding to assessment prompts embedded in a digital game or 
simulation, can perform a wide range of actions continuously over extended periods of time leaving behind 
high-resolution traces of decision-making, intentions, and even emotions in a new kind of digital 
performance space (Dede, Nelson, Ketelhut, Clarke, & Bowman, 2004; Gibson, 2011). Large files often 
comprising thousands of records for a single virtual performance interaction are generated. Interpretation of 
issues of time, sequence, action relevancy, big-data pattern recognition, overlapping patterns, and levels of 
meaning is required (Ifenthaler, Eseryel, & Ge, 2012). There is a need for new psychometric considerations 
for methods of data capture, analysis and display. 


3. CLASSROOM MANAGEMENT IN SECOND LIEE 

Actual experiences and the informal relationships within the school classroom settings can lead to a sense of 
belonging to a community of practice thus building the aforementioned and requisite self-efficacy (McNally, 
Blake, & Reid, 2009). As part of a course embedded in a field-based internship students examine personal 
beliefs in relation to various methods of classroom management. Student teaching follows where students 
are expected to enact those beliefs in a real classroom. Initial data reflects students’ genuine investment in 
becoming effective classroom managers with many references made to wishes that pupils learn, enjoy the 
experience of learning and feel safe. Virtual practice in virtual classrooms should increase students’ 
efficaciousness paving the way for successful early teaching efforts because they are coming very close to 
having authentic teaching and collegiality building. 

3.1 Process 

Several colleagues at a large southwestern university launched a virtual third grade classroom; a prototype to 
ascertain whether the virtual reality tool. Second Life, would be a viable means to help pre-service teachers 
practice classroom management skills in a relatively risk-free environment. Over the course of nine months a 
team created an inviting virtual classroom in which instructor-created scenarios were translated into a virtual 
classroom. Students described and analyzed their learning after participating as either a student or a teacher 
and co-constructing new points of view collectively. 
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3.2 Student Intern Responses 

After each scenario enactment debriefings were held. Everyone simply chatted about the virtual experience. 

In turn, students made personal connections, outlined new understandings, analyzed, questioned, and 
critiqued themselves in formal written papers. Analysis of this data gave evidence that students' integrated 
knowledge of teaching from the perspective of either the actors (student and teacher) or the audience. Actors, 
as avatars, were able to be the role they were playing and also retain a distance from that role (Ackermann, 
2004). 

Analysis of transcripts of the debriefings and student reflections revealed themes related to the goals of 
the course including: ways a mentor manages a classroom, attention to positive behaviors, organization, 
teacher control, proactive teacher behaviors, consequences for misbehavior, chaos, choice, humiliation, 
respect for students, hearing students, attention to non-demanding students, student engagement, humiliation 
of students, and knowing ones’ students. 


4. VIRTUAL PEDAGOGICAL PRACTICE IN SIMSCHOOL 

A major challenge facing beginning teachers is how to juggle teaching and learning parameters in an 
often-overwhelming context of a new classroom. The classroom simulator, simSchool, has been developed, 
improved and researched for its role in contributing to teacher development to help address this challenge. In 
this brief overview, we provide the rationale for using a computational model as the engine for a teaching 
simulation, describe in broad terms how the model works, and share the results of research on simSchool. 

The use of digital games and simulations to help prepare teachers is inspired by the dramatic rise and 
growing appreciation of the potential for games and simulation-based learning in professional training 
(Aldrich, 2004). Research and development of teacher education games and simulations is just beginning. 
The new field has the twin goals of producing better teachers and building operational models of physical, 
emotional, cognitive, social and organizational theories involved in teaching and learning (Gibson, 2009). 

4.1 Computational Representations of Teaching and Learning 

SimSchool promotes pedagogical expertise by re-creating the complexities of classroom decisions through 
mathematical representations of how people learn and what teachers do when teaching. The model includes 
research-based psychological, sensory and cognitive domains similar to Bloom’s Taxonomy of Educational 
Objectives (Bloom, Mesia, & Krathwohl, 1964). However, in simSchool these domains are defined with 
underlying subcategory factors that reflect modern psychological, cognitive science and neuroscience 
concepts. Eor example, the Eive-Eactor Model of psychology (McCrae & Costa, 1996) serves as the 
foundation of the student personality spectrum. This model includes the following characteristics: 
extroversion, agreeableness, persistence, emotional stability, and intellectual openness to new experiences. 

Aspiring teachers interact with this cognitive model over several sessions spanning several weeks, with 
micro-teaching interactions lasting from 10 to 30 minutes; and attempt to negotiate the simulated classroom 
environment while adapting their teaching to the diversity of students they face. Additional details 
concerning how simSchool works - how the simulated students respond to tasks and teacher talk - can be 
found in (Christensen, Tyler-Wood, Knezek, & Gibson, 201 1). 

4.2 Indications from simSchool Research 

Results spanning several years of study (Knezek, Eisser, Gibson, Christensen, & Tyler -Wood, 2012) suggest 
that simulations such as simSchool can play an important role in preparing tomorrow’s teachers. Use of 
simSchool has been shown to increase Instructional Self-Efficacy (confidence in one’s competence), Learner 
Locus of Control (the teacher’s sense of responsibility for learning results), and Self-Estimates of Teaching 
Skills, Experience and Confidence. 
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Research on SimSchool indicates that it provides pre-service teachers with a safe environment for 
experimenting and practicing techniques, especially methods of addressing different learning characteristics, 
and wide variations in academic and behavioral performance of students. Findings reported to date illustrate 
that simulations are capable of modeling a wide range of student learning, and can be envisioned as having 
significant impacts on improvements in teaching. 


5. SIMILARITIES AND DIFFERENCES 

simSchool and Second Life enactments of the teaching act have some similarities. In both models, students 
play the role of teacher interacting with virtual students. Both technologies support the acquisition of 
confidence in one’s expertise as an actor who can practice making effective instructional decisions without 
risk of harm to real children. simSchool students work with virtual students in a virtual classroom with 
performance and social characteristic student profiles, and during “run time” make instructional decisions. 

The primary difference between the two models lies in how knowledge is constructed while students are 
in situ. In the Second Life students are both the actors and the acted upon in concert with an audience of their 
peers. Students construct their understanding of effective teaching socially before, during and after the 
experience. In simSchool a replay capability allows students to experiment to find better moves to assist 
virtual pupils by adjusting tasks, task sequences, and personal interactions. Thus, knowledge of teaching 
practice is socially constructed in Second Life; whereas, this knowledge is privately constructed through 
experimentation with a computational model of teaching and learning in simSchool. 

In simSchool the progression of the scenarios are repeatable within close but flexible bounds, allowing 
for single player experimentation and private asynchronous practice. In Second Life, the scenarios are 
socially constructed and are experienced in a live, real-time, group setting. 

simSchool allows pre-service students to create virtual students with particular psychological and 
cognitive characteristics, build a virtual classroom with targeted performance and social characteristic student 
profiles, and during “run time” make instructional decisions. In Second Life, however, students inhabit 
pre-constructed avatars ascribing personal character during the act of role-play, much like improvisational 
theatre. 


6. DISCUSSION 

The delivery environments for Second Life versus simSchool differ somewhat. Second Life is a very large 
system serving multiple functions and numerous islands, while simSchool is a single -purpose environment. 
Instructors need the support of an Instructional Technology team to create a virtual environment in Second 
Life. In simSchool, the complexity of the modeling framework and its openness for flexible purposes entails 
building up experience in applied educational psychology and cognitive science as well as teaching methods, 
which provide benefits for effectively creating scenarios and making sense of the data produced by the 
simulation. 

New assessment challenges and opportunities abound in both the role-playing virtual worlds and digital 
simulations contexts. While many of the ensuing questions are common (e.g. to what extent these experience 
have an impact on one’s preparation to teach, cause valuable conceptual changes, and improve one’s 
knowledge, skills and attitudes as a teacher) the approaches differ for addressing them, leading to new 
methods of inquiry and analysis. 

Role-playing as a teacher in Second Life when combined with face-to-face instruction invites the 
researcher to consider the long-term behavior effects on pre-service teachers’ future teaching practice. Virtual 
reality allows the learner to be immersed in the environment with a high degree of authenticity. By 
combining real-life problems of great intrinsic value to the learner with a dedicated space designed for the 
learner to solve those problems with cooperative support of their peers and instructor, learners are required to 
analyze their choices and those of others. The question becomes whether any of this results in deepened 
conceptual understanding and behavior change leading to effective teaching practices. 

As indicated earlier, digital games and simulations based in computational models provide learning 
experiences in a dynamic new performance space that have implications for assessment. Data mining and 
machine learning in a context of data science applied to educational measurement are approaches and theory 
needed to address the new challenges of assessment (Gibson & Knezek, 201 1). 
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7. SUMMARY AND CONCLUSIONS 

Two alternative technologies forming the basis of computer-mediated teacher preparation systems have been 
compared and contrasted regarding implementation, operation, and assessment considerations. The role 
playing system in Second Life is shown to have the unique characteristic of developing shared, constructed 
pedagogical knowledge, while the flight simulator metaphor of simSchool encourages rapid, stepwise 
refinement of pedagogical expertise. Each has cost and traveling distance advantages over face-to-face 
traditional meetings, as well as shortcomings. Ultimately, the largest assessment issue for both is how to 
measure learning inside a simulator or a social media space. Further research is needed in this area. 
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ABSTRACT 

This paper discusses some changes in the context of reading brought about by new forms of text publishing and reading, 
such as PDF files, websites and eBooks. In order to do so, the paper starts with some discussion on the importance of such 
devices. Then explores some of the main their characteristics, as well as some consequences it might bring to the behaviour 
of readers. It ends with some observations on how this new perspectives of regarding have been dealt with in the Brazilian 
educational system. 
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1. INTRODUCTION 

Nowadays, books, newspapers, academic journals and all sort of publications are becoming digital. They are 
now likely to be part of the same reading reality as other ordinary webpages: as we browse and read these them, 
we might have multiple tabs open, click on diverse information in order to check and get more accurate data. As 
a result, depending on the occasion, our background on a given subject is now an on-going process. 

According to Lemos (2012), the sales of electronic books (eBooks) have already outgrown the actual 
“physical books” in many online bookstores (such as Amazon). As a result a number of reading devices is 
increasing; besides Kindle, one might find a number of devices such as Kobo, a Japanese eBook format. Nook, 
sold by Barnes & Noble among many others. This brought about a new reality, reading devices demand a 
materiality which is rather different from computers, tablets and printed materials. 

During the last 30 years, major technological changes have led to a great change in the way new we relate to 
language (KRESS, 2003) and, as a result, to the text. This includes not only the way texts are produced, 
assuming syntactic, lexical and semantic changes, but also changes in text support: in many cases text has 
ceased to be printed. As a result, computer monitors and locative devices, such as phones and tablets, are 
becoming a natural place for reading and writing. It is now important that we start discussing new theories and 
methods in order to make the teaching of reading closer to everyday practices. Actually, media research needs 
to take us to discussions on media digital literacy in order to bring them to schools and to students’ daily 
practices, making them more meaningful to the many social contexts (Buckingham, 2008; Buckingham, 2010) 
our students live in. 

On the one hand, such new technologies have allowed the expansion of our universe from a reality which 
was merely typographical to another that takes many other semiotic resources into account. On the other hand, it 
has also ensured our contact with a variety of information that was once unimaginable. Network technologies 
arise with the unique ability to aggregate individuals and contexts in order to promote a process of reintegration 
of a human diaspora (Levy, 1998). Naturally new forms of text play a significant role in this argument, since 
they continuously expand the possibilities of meaning making of a text (Bolter, 2002). The practical result of 
such transformation is the birth of a series of new content and social practices. Firstly, it becomes important to 
perform studies that focus on how such contents are received in different communities. In this sense, it would be 
important to develop research to help us understand the process of structural configuration and reconfiguration 
of the interaction amongst users, media and resources. 

This paper aims at bringing about an initial discussion on some possible changes in the behaviour of readers 
in three platforms: PDF files, eBooks and websites. The discussion looks at how the technological features of 
such platforms may change reading strategies as the technical elements of each of these platforms are presented. 
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A reflection on such reading process is also presented. The final remarks tend to be centred in educational 
changes that might take place in the Brazilian context. 


2. SOME CHANGES IN READING STRATEGIES 

Most reading strategies adopted for screen reading are rather different than those used for printed texts. Due to 
the great amount of information present on webpages, PDF files and eBooks, readers tend to read vertically, 
constantly skimming and/or scanning the text in order to get a general comprehension of its structure and to 
grab the specific pieces of information that are relevant. Besides that, those readers rarely read an entire website, 
they tend to follow links and move on as soon as more interesting aspects comes along. In fact, as Lemos (1998) 
mentions, most of the knowledge developed in this interaction tends to be superficial, that is, online readers tend 
to connect to a greater deal of information, but most of the time their attention is kept on a superficial level. This 
kind of reading strategy is similar to patchworking, the many paths possibilities allow readers to create their 
own texts. Therefore, these texts become a result of the interaction of the many other texts. The result is 
something highly personal, since there might not be equal textual experiences, that is, not all readers follow the 
same path through reading from the Internet. 

Nowadays PDF files have evolved to an encapsulating format that allows not only images, but also videos 
and audio and web -links to be placed just within a single text. This not only makes PDF resemble hypertexts, 
but also attaches an important feature to this device, since it does not depend on a network connection: video, 
audio and internal clickable links - which usually refer to pictures, tables etc. - are embedded into the file. In 
these texts each textual reference might be followed,. This is a textual phenomenon that used to be possible only 
in websites. 

However, this seems to be a much more endogenous kind of reading. In other words, if a PDF file does not 
offer links to websites, the reader navigates only within the text. Naturally it also provides a way of reading, 
which is rather different from traditional books. The possibility of using such resources allows the reader to 
create a pathway that is similar to hypertexts. Due to the possibilities offered by such links, the PDF file is read 
in an order that is rendered regarding the interest of the reader to follow the text’s internal references. 

EBook reading devices. Kindle being the most popular format, are different. They seem to be a much 
simpler file format and its emphasis is elearly on the “contenf ’ of a text, rather them on its form. They do not 
encapsulate all the semiotic systems and elements that we have in websites and PDF files, in most of the cases 
they seem clear text files with a fixed typesetting and no colour. Nevertheless, Kindle eBooks are much lighter 
files and are easily distributed through 3G networking available to certain readers, or Wi-Fi, which is the 
standard technology of this device. 

If we think in terms of reading strategies, some important changes might operate. PDF files offer some 
important resources for skimming through the text, thumbnails can give us a general picture of the way the 
pages look like. Here the pages are available in the right side of the screen in reduced size. The reader might 
choose which page to click and then read it in whatever order s/he likes, some programmes even allow the 
rearrangement of the pages to an order that suits better the readers’ reading purpose. In eBooks the possibility of 
readers changing text order does not exist in this type of reading, browsing does not even follow the concept of 
page. Kindle books, for example, are organised according to position, that is, the portion of the text that fits into 
the screen. Position might also be variable according to the size of the font that appears on the screen. If the 
reader chooses a larger font size, there will be less text displayed on the screen, therefore, the position is never 
the same for every reader. For example, a reader who uses smaller font size, will therefore, see more lines on 
the screen In the latest Kindle versions, the interface tells us in which page we are in reference to some printed 
issue; this reference, though, does not show us in where each page starts or ends. EBook skimming seems to 
become a rather sequential matter; we actually have very few possibilities to navigate as we have in printed 
books. 

Automatic search is standard out-of-the-box technology in any word processor, PDE or eBook reader. They 
allow searching for regular expressions, which may be composed by single letters or even a whole sentence. 
Such tools, in some sense, substitute the need for scanning the text using only our naked eyes. The discrete 
element that is named the search-target is easily found and, sometimes, clearly highlighted. It is rather different 
from traditional printed reading, where the scanning depends only on both reader’s ability to establish subjective 
and objective search criteria as well as his/her capacity to scan through the text. 
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This is the reason why this technology is well used by readers who believe that specific lexical items or 
expressions might be present in a given text. If used well, it might be a very important tool to help students to 
check some predictions regarding the contents of what they read. It might be also a very productive way to 
make our students to pay more attention to the words which are around their search node, teaching them that 
any word meaning is, to some extent, defined by the language choices that are put together in texts. 

Two of the platforms discussed here (PDF and eBooks) enable the use of notes, which allow the 
highlighting of some part of the text. For PDF files, some specific programmes are needed to perform this 
action. It is so because some standard Adobe readers do not allow readers to annotate in documents which have 
DRM protection. For eBooks, this is a standard feature without any restriction. For some readers, such notes are 
also saved in *.txt file, which can be read separately. It is a tool rather useful for later study or even for 
comparison between student’s notes. 

PDF files, carry a lot of formatting and visual information. This type of file is, sometimes, a facsimile of 
printed material. In eBook readers, on the other hand, texts do not have any typographical appeal, even the use 
of images here seem to be very limited, that is,, most of the readers are monochromatic and do not integrate 
images as PDF files do. Most of the times, images stand alone on the screen. 

EBook readers and some PDF readers have interesting feature: if you select a word in a text, you can look it 
up automatically in a dictionary. Different models have singular procedures, but in most of the cases the readers 
only need a click or two. In the touch screen model, for example, the reader only has to hold his/her finger on a 
word for a dictionary definition to come up. In simpler models, the word must be selected and a secondary 
menu shows up. These kinds of facilities are important for integrating the dictionary as a reading tool within a 
same device, this allows the reader to clarify his/her doubts on the spot with no time wasting. This might lead to 
an overuse, and some dependency on dictionaries. 


3. SOME REMARKS ON THE BRAZILIAN CONTEXT 

A consequence of this new type of technological reality is the need to establish a reading instruction programme 
that aims at teaching techniques regarding digital reading. In the case of Brazil, the issue is complex and 
involves several factors. Despite of the fact that several schools have access to computers, most of them do not 
have access to eBook readers. It makes difficult the access to equipment. It is important to notice that much of 
what is discussed here would lead to a teaching of reading that would rely on strategies adapted to digital 
reading. Unfortunately, in the Brazilian context, this component is not part of the teachers training curriculum 
programme in the vast majority of national Universities. The Access to the Internet and telecommunication 
networks might be another problem: notwithstanding the wide popularization of these elements in Brazil, the 
Internet is still accessed by only a little over 50% of the population. This is a determining factor for reading in 
any of the digital models discussed above. Regarding classroom practice, the teacher would not only be teaching 
technical procedures of each of the hardware or software discussed here, but s/he would also be able to bridge 
these techniques to the reality of students' reading practices. Brazilian schools are indeed still lagged in relation 
to student’s digital reading habits. At schools, the existence of this reality is simply ignored. The vast majority 
of computer centres is used just as research engines, and the overwhelming majority of schools do not have 
access to tablets. Even teachers, as already mentioned, have little or no experience of the platforms investigated 
here. 
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ABSTRACT 

This reflection paper challenges current test scoring practices on the grounds that most wrong-answer selections are 
thoughtful not random, presenting research supporting this proposition. An alternative test scoring system is presented, 
described and its outcomes discussed. This new scoring system increases the number of variables considered, reduces the 
mesh of the analytical screen and provided considerable more information to inform teaching. 
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1. INTRODUCTION 

This paper takes a critical look at current multiple -choice test-scoring practices. Consider Dr. Friedrich A. 
von Hayek’s (1974) Nobel Prize acceptance lecture “The Pretense of Knowledge.” 

There is much reason to be apprehensive about the long run dangers created in a much wider field [than 
economics] by the uncritical acceptance of assertions which have the appearance of being scientific [because] 
what looks superficially to be like the most scientific procedures are often the most unscientific ... Beyond 
this, there are definite limits to what science can achieve [arising] from our inability to quantify some 
important variables. 

This paper quantifies these intractable variables in a novel manner. 


2. THE “RIGHT-WRONG” SCORING PERSPECTIVE (LEARNING 
HYPOTHESIS I) 

We currently presume that multiple -choice test scores estimate what students know as a cumulative 
proportion of an epistemological domain. The items are a representative sampling of course content. The 
frequencies of “righf’ answers are assumed proportional to the examinees’ “knowledge.” Options are often 
designed to reflect common procedural errors or common misconceptions (e.g. concept inventories; Halloun, 
I. &Hestenes, D.L. (1985)). 

Most “wrong” answers are randomly selected and unrelated to the “right” answers (making scoring into a 
dichotomous process.) The frequency of “right” answers is necessary and sufficient information base about 
examinees’ subject-matter knowledge. 

An Editorial in Science (Coffey & Alberts, 2013) suggests that: 

“The [Common Core] contains a vast number of core disciplinary ideas and subideas. 

Current measurements and approaches do not allow these performances to be assessed 
easily...” 

In this approach we can expect a number of supporting observations. The most important of these are: 

1. The distribution of scores will resemble a typical normal bivariate distribution (Frame 1). 

2. The changes of answer selections from one administration to the next (whether the same test or a 
parallel one) will be from any “wrong” answer to the “right” one (Frame 2). 
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3. That changes of answer and direction of learning increase will progress in equivalent directions 
{the equivalency assumption'. Frame 3). 

Current practice presumes there can be only one reasonable “right” answer for a question. Learning 
requires the exchange of “wrong” answers for “righf’ ones (Frame 1). Answering is a dichotomy whose 
distribution of total frequencies resembles the upper limit of the binomial expansion (a + b)" as n^oo. This is 
the normal bivariate distribution (Frame 2). The only meaningful changes are those in the total-correct 
scores. (Frame 3). 


3. “MOST ANSWER SELECTIONS ARE THOUGHTFUL” (LEARNING 
HYPOTHESIS II) 

Education suffers non-measurability as much as any other discipline. We can compare live data with the 
hypothetical patterns postulated in Hi. Which theory stands the test of oservation. If the results from live data 
show patterns different from the expectations for Hi, it must be rejected in favor of Hn? 


4. WHAT DO MULTIPLE-CHOICE TEST ACTUALLY MEASURE? 

Extending over more than half a century, interviewing students about their “wrong” non-random answer 
selection distributions (Erame 4), using thousands of students, a broad age range and more than one continent 
and more than one subject matter area showed item interpretation to be involved with the following 
observations: 

1. Teaching interpretation skills causes huge increases in students’ learning motivation (Powell, 
2010a). 

2. Using written selection explanations with adults, Powell (1968), showed explanations often 
predicted selections, 

3. Testing 550 students (with Gorham, 1956) from the third through the eighth grade, showed (Erame 
4; Powell, 1977) “wrong” answer subsets systematically ordered perfectly with chronological age 
(CA; Erame 5). More than half these correlations are conjoint (double-headed arrows; Luce & 
Tukey, 1964). 

4. Giving this test twice (October-March) to 2,000 h- students (third grade through the end of high 
school) supported this developmental sequence. We defined students who followed the observed 
developmental sequence as showing increasing cognitive maturity (CM). This study showed four 
developmental pathways, refuting the dichotomous data assumption and displaying the dynamics of 
learning (Erame 6): 

a. A Piaget-like (1953) normal development (about 40% of the students), 

b. Declining CM (about 30% of the students), 

c. Students stalled at the literal (Concrete Operations) thinking level at a preadolescent level 
interpretation of questions (about 20% of the students) and 

d. A mature age multidimensional expansion of thinking skills beyond 2-value logic. Shown 
by shifting from a “right” answer to a “wrong” one (the remaining 10% of the students). 

5. Using a representative sample (52) of these about 3,000 students, we drew developmental profiles of 
their CM changes (Erame 7), which shows a student whose CM declined while total-correct score 
increased and a student with the opposite pattern of change. These two contrary-to-expectations 
results are sufficient to refute Hi. Powell and Powell showed that the 10% can be increased to at 
least 40%. 

6. Using 16,000 h- students from years 4, 6 and 8 in India and two tests, one in science and one in 
mathematics, Powell, Bernauer and Agnihorti (2012) showed that item interpretation was the key 
ingredient for answer selection with different tests. It showed developmental sequences, procedural, 
information style and cultural bases for answer selection of “wrong” answers (No Frame). 

7. Using the multidimensional transition to identify high order changes among students Powell and 
Powell (2012) showed that when teaching interpretation skills replaces transmitting information a 
six-fold increase in multidimensional thinking among college undergraduates occurs (No frame). 
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8. From a representative sample of the data from study 4, Powell, (2013) showed (Frame 8), the 
changes in cognitive maturity (CM) are unrelated to changes in total correct score and that the latter 
score misclassified more than two thirds of the students from the corresponding results drawn from 
the same test. (Compare this result from live data with the anticipated result from random “wrong” 
answering given in Frame 3). Current scoring practice is invalid! 


5. CONCLUSIONS AND IMPLICATIONS 

When including all answer-selection variables, multiple-choice tests measure interpretation skills 
independently of knowledge of the subject matter content of the questions except that fluency with the 
technical language of the item is partially involved. These skills are not dichotomous. Instead they can be 
expanded into logics of levels higher than 2 and this shift is observable using the Thurs (). Adding the 
“wrong” answer variables to the mix makes these data multinomial, not binomial, invalidating current 
test-scoring practice at the expense of two-thirds to three-fourths of the data available from each student from 
a single administration of any test and more than this when the tests are administered more than once to 
expose the learning dynamics. When “wrong” answers are omitted from the scoring process, this loss of data 
destroys the integrity of the matrix. 


6. THE FIX 

The algorithm for calculating the Thurs ()[/>] (Frame 9), that bypasses linear dependency and detects non- 
linear data structure, is now in public domain for private use. The interpretation strategies found from 
interviews data from the 1977 study and the data samples from the 1992 and the 2013 studies may be 
requested from the author. The author will consult with any interested party on application of the p or 
educational effective improvement strategies. 
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ABSTRACT 

The purpose of this paper is to introduce a design model for supporting student volition. Eirst, the construct of volition is 
explained and the importance of volition is further described in the context of goal attainment. Next, the theoretical basis 
of the model is described. Last, implications of the model are discussed for the design of learning environments as well as 
teacher education. 


KEYWORDS 

Volition, emotion regulation, action control, motivation, design 


1. WHAT IS VOLITION? 

Volition refers to a psychological power that allows people to persist during the process of goal striving 
(Heckhausen, 2007; Kim & Bennekin, In press). Students with volition are persistent in working on learning 
tasks even when encountering difficulties, challenges, distractions or temptations. 


2. IS VOLITION DIFFERENT FROM MOTIVATION? 

Motivation is only desire. When difficulties, challenges, distractions or temptations arise, desire may be 
insufficient to enable students to persist (Gollwitzer, 1999). Desire simply initiates the process of attaining a 
goal and volition helps follow through to actually attain the goal (Heckhausen, 2007). 


3. WHY IS VOLITION IMPORTANT IN LEARNING? 

Volition can help students (a) take actions toward goals, (b) sustain motivation, and (c) control emotions 
(Kim & Bennekin, In press). First, as stated earlier, students with volition follow through because their 
intentions are implemented (Gollwitzer, 1999). Second, when motivation withers, volition can reconcile 
insufficient motivation (Kehr, 2004). For example, when a student’s motivation to study for an exam 
weakens because of his newer desire to be with his friends partying, volition would allow him to be in a 
physically distant place and compensate for his weakened motivation. Last, students with volition can 
manage unwanted emotions (Corno, 1993). Emotion control is part of volitional strategies (Kuhl, 1987) and 
crucial in learning processes and outcomes because emotions can alter not only motivation but also memory, 
decision making, and use of cognitive strategies (Pekrun, 2006). 


4. VOLITION SUPPORT (VOS) DESIGN MODEL 

Not every student is volitional but volition can be cultivated (Byman & Kansanen, 2008). This paper 
proposes a model for designing support for student volition (e.g., creating volition support into existing 
instruction, designing external tools). 
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The model is built on theories and research not only on volition but also on motivation and emotion 
regulation (e.g., Corno, 1993; Gollwitzer, 1999; Gross, 2008; Keller, 2008; Kuhl, 1987). The model consists 
of the four stages: (1) goal initiation: “Want it”, (2) goal formation: “Plan for it”, (3) action control: “Do it”, 
and (4) emotion control: “Finish it”. 

First, the goal initiation stage is to help students see the value of learning tasks and desire to set goals. As 
discussed earlier, a desire is necessary to initiate the goal pursuit process that requires also volition. Second, 
the goal formation stage is to help students formulate goals and plan what to do, with two emphases. One 
emphasis is on guiding the content of students’ goals to be specific than general and proximal than distal as 
well as to promote positive outcomes than negative outcomes and mastery than performance (Gollwitzer, 
1999). The other emphasis is on specific planning for implementation intentions such as writing down a list 
of tasks to do as well as a list of plans in the “if’ and “then” form (Gollwitzer & Sheeran, 2006). Third, the 
action control stage was to help students to actually do what they said they would do by asking them to 
manage resources and monitor their implementation intentions. This stage involves reminding students of 
their goals, tasks, and plans, showing them strategies to manage resources, and having them reflect and 
monitor their progress in terms of task completion and goal achievement. Last, the emotion control stage is to 
help students finish what they said they would do for study while regulating their emotions. Strategies 
offered in this stage are about how to modify surroundings, select different environments, shift attentions, 
conduct cognitive reevaluation of situations, and suppress activated emotions (Gross, 2008). 


5. DISCUSSION 

The VoS design model provides a synthesis of the literature on volition as well as motivation and emotion 
regulation. The model also provides directions to the researchers and practitioners who are interested in how 
to help students persist in their learning processes. The model can be applied not only to create learning 
environments but also to teacher education that helps teachers recognize the importance of their role in 
cultivating student volition and learn how to do so. Theoretical foundations for the model, strategies for each 
stage of the model, and practical applications of the model in design will be presented in detail at the 
conference. 
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ABSTRACT 

Most electronic communities have little crossover communication with each other, beyond specific linkages to other 
sites. So there is scant crossover among the world of academia, the “world of work,” and the virtual world of social 
media. The objective of this research is to examine e-leaming communities across academic disciplines to explore 
“crossover” research in multiple academic disciplines, as well as in eleaming and adult learning, to see if there could be 
key issues brought to the foreground that affect adult learners and student retention. In addition to conducting historic 
research across a number of academic disciplines, the study also provides a case study that could reveal insights into 
factors that shed more light on adult learners and retention in online courses by traversing academic disciplines to achieve 
thicker research and contribute to the body of research on online adult learners in a range of academic disciplines. 
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1. ACADEMICS’ HESITATE CROSSING ACADEMIC BORDERS FOR 
RESEARCH THAT COULD HELP ADULT ELEARNERS 

The mass availability of global electronic databases, however, now allows researchers to stumble upon expert 
research studies from other academic disciplines that could shed new light on old questions, such as how to 
help students complete their academic studies. 

What does research across academic disciplines have to do with adult learners and student retention? 
While most universities budget monies and resources for new technologies to accommodate social media, 
few are looking at ways to restructure students’ ways of learning new materials with those technologies and 
what they already know. Prior knowledge could be important, particularly because, as of 2013, students are 
older than traditional age students of the average ages 18-22, and they come to college with a vast array of 
experiences. While traditional age students still predominate at four year universities, there are a growing 
number of students who are generally older, averaging 23-27 years of age. About 43 percent of 
undergraduates enrolled in postsecondary education in the United States are age 24 or older (National Center 
for Education Statistics, 2007). These adult learners, because of the additional responsibilities and time 
requirements, often choose online courses. But the course completion rates and graduation rates of students 
enrolled in online courses are much lower, averaging 60% completion of online courses, and slightly lower 
completion rates for online programs (Gallop, 2013), so, the question is, with more students being attracted to 
online courses, how can administrators help students to have higher completion rates in those online courses 
and programs? The problem of attrition in online courses persists, regardless of academic discipline, so it 
would be helpful if there may be best practices found that could provide insights into higher student retention 
across academic disciplines. 

The researcher in this study first researched data available on distance and e-learning in the disciplines of 
computer science. Education, Commerce, and Human Sciences then found there was also a growing body of 
research on distance learning as a separate field of research. 
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She then sought to integrate these subjects related to successful course completion and student retention 
in online courses in those disciplines. Computer Science and Software Engineering on eLearning, Education, 
Curriculum, and Instruction on eLearning, Commerce, and distance education as an academic discipline. But 
there are important research gaps in all of the above-mentioned fields of study, particularly with respect to 
eLearning and social media. Perhaps the reasons for the gaps could reside, at least partially, from the 
segmentation of all these arenas, none of which, ironically, are segmented when a person conducts an 
Internet Google or Wikipedia search. 

The researcher examined research in the above academic discipline areas; she also looked at issues of 
cognitive load and its possible effects on students in online courses; and she conducted case study research to 
see if interdisciplinary research could provide more insight into how to help students navigate successfully 
through online programs. The explorations in this research provide data that can encourage academics to 
traverse broader intellectual plains; they can find these new “category” borders readily available simply by 
crossing over into a wider array of disciplines by conducting an Internet search; those who elect to span 
multiple academic and cultural disciplines can find vast new territories of knowledge bases, like treasure 
troves, waiting to be discovered. 
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ABSTRACT 

Parent-child relationships may be strengthened when parents and children play video games together. Literature is limited 
in addressing the impact of co-playing video games on parent-child relationships. Eamily systems theory, in particular, 
parental mediation through co-play, may provide insights into parent-child relationships. Parents who co-play in order to 
mediate video game content or child behaviors are selecting one of the less restrictive forms of mediation. Co-playing is 
likely to have an impact on parent-child relationships. Additionally, girls may benefit more than boys from co-play and 
fathers as co-players may have a greater impact on relationships than mothers. 
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1. INTRODUCTION 

The family-centered video gaming movement can be traced to the 2006 Nintendo Wii (Chambers, 2012). The 
trend is believed to be brought about in part by parents’ concerns about social disengagement of their video 
game playing children, the mobility of devices, and the near-constant access children have to video games 
making parental monitoring a challenge (Chambers, 2012; Jiow & Lim 2012). In an effort to address these 
concerns, some parents have taken to playing video games with their children (Coyne, Padilla- Walker, 
Stockdale, & Day, 2011; Eklund & Bergmark, 2013; Nikken, Jansz, & Schouwstra, 2007; Shin & Huh 201 1). 
There is limited literature on the impact of co-playing video games on the relationships between parents and 
children. A review of the literature is needed to identify gaps in the research and make recommendations for 
future study. 


2. THEORETICAL FRAMEWORK 

In this paper, co-gaming is examined through the lens of the family systems theory, which is one of the most 
influential theories in family sciences (Charles, 2001). The theory states that an individual can be best 
understood with an understanding of the family system to which that individual belongs because “family 
systems theory is concerned with family dynamics, involving structures, roles, communication patterns, 
boundaries, and power relations” (Rothbaum, Rosen, Ujiie, & Uchida, 2002, p. 329). The family systems 
theory was developed for use in clinical therapy, but has recently been integrated with nonclinical theories. 
Researchers studying gaming within families have used the family system theory as a theoretical framework 
(Buswell, Zabriskie, Lundberg, & Hawkins, 2012; Padilla -Walker et ak, 2012). 

2.1 Characteristics and Behaviors of Co-gamers 

According to Nikken and colleagues (2007), parents who play video games with their children tend to believe 
that games can have a positive effect on children. Additionally, co-playing parents are usually younger and 
have more experience personally playing video games than those who don’t co-play (Nikken et ak, 2007). 
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In fact, 81% of surveyed parents who were active gamers reported playing video games with their 
children (Nielsen, 2008). However, the co-play of non-gaming parents was significantly lower. Only one 
third of adolescents sampled in a separate study reported having played with a parent (Padilla -Walker et al., 
2012). There was no significant difference between the amount that mothers and fathers play (Nielsen, 2008). 
Literature describes contradictory results comparing the co-play with sons and daughters. Two study showed 
the amount of co-play with sons and daughters did not vary (Coyne et al,, 2011; Padilla- Walker et al., 2012), 
whereas another study related that parents played more with sons (58%) than with daughters (37%) (Ipsos 
MORI, 2009). This could be due to differences between sample populations. The video games co-play 
generally lasts between 30 and 60 minutes per session at a frequency of a few times per week (Ipsos MORI, 
2009). The games played by parents and children are primarily the “active techno logy/fitness games” genre 
followed by racing games, and then educational games (Ipsos MORI, 2009, p. 18). 

The primary reasons that parents reported for co-playing games with children were that their children 
enjoyed it, their children asked them to play, they believed co-playing would improve the children’s 
cognitive skills, they wanted to monitor the content of the games, and that co-playing allowed them to spend 
time with their children (Ipsos MORI, 2009; (Nielson, 2008). The primary reasons that children cited for 
playing with an adult family member were that it was more fun and that it allowed them to spend time with 
the adult (Ipsos MORI, 2009). Interestingly, among children who don’t play with a parent, the children 
reported the top reason against co-playing was that it is less fun to play with the parent. Parents who don’t 
co-play video games with their children state that they don’t enjoy playing the games (Ipsos MORI, 2009). 

2.2 Co-playing Video Games as Mediation 

Parenting styles can impact the parent-child relationship. As such, the form of mediation used to manage 
video game playing can offer insight into the relationships between co-playing parents and children. Parents 
use a variety of methods to mediate video game playing with children including active mediation (i.e., 
discussing negative and positive game attributes and player behaviors), restrictive mediation (i.e., stopping 
gameplay), and parent-child co-play (Nikken & Jansz, 2003, 2006; Shin & Huh, 201 1) to guard their children 
from the potentially harmful effects of video games. Some of the reported behaviors that were measured 
include frequency of video game play (Nikken et al., 2007; Shin & Huh, 2011), pro-social behaviors (Coyne 
et al., 2011; Shin & Huh, 2011), deceptive behaviors (Shin & Huh, 2011), and internalizing and aggression 
(Coyne et al., 2011). Parental mediation of decreased slightly as teenagers get older (Shin & Huh, 2011). 
While co-playing was perceived by some parents as an effective mediation strategy (Shin, Huh, 2011), it has 
been suggested that, due to parents’ limited time, co-play is not used as frequently as more convenient, 
restrictive mediation (Nikken & Jansz, 2006). 

2.3 Co-playing Video Games to Strengthen Relationships 

A small number of studies have addressed the effect of co-playing video games on relationships. The gaming 
industry promoted co-gaming as a way to “foster family harmony” at a time when parents were concerned 
that video game playing was distancing children from the rest of the family (Chambers, 2012, p. 37). Both 
parent and child-reported connectedness was higher when parents co-played video games with their children, 
particularly in girls when playing age-appropriate games (Coyne et al., 2011). Padilla-Waler and colleagues 
(2012) suggested that the co-play represent a shared interest between parent and child, thus strengthening the 
relationship. Regular game co-playing between a father and child was found to improve family functioning, 
which would indicate stronger relationships (Buswell, Zabriskie, Lundberg, & Hawkins, 2012). 


3. CONCLUSION 

The limited literature addressing parent -child relationships through game co-playing indicates that co-playing 
may positively impact family relationships. Parents who co-play with their children are exhibiting one of the 
less restrictive forms of mediation, which may positively impact parent -child relationships. Girls who co-play 
may be affected more than boys (Coyne et al., 201 1), and co-playing fathers may have a greater influence in 
improving relationships than co-playing mothers (Buswell et al., 2012). 
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Parents who play with their children should be aware of the potential for unintended consequences of 
co-playing games with violent or sexual content. Playing these games may result in increased interest in such 
content and more frequent playing of such video games with friends. There is a need for more research in this 
area to contribute to our understanding of this family experience. 
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ABSTRACT 

A ubiquitous English vocabulary learning system: evidence of active/passive attitudes vs. usefulness/ease-of-use 
introduces and develops “Ubiquitous English Vocabulary Learning” (UEFL) system. It introduces to the memorization 
using the video clips. According to their paper the video clip gives better chance for students to memorize vocabulary. 
There are few positive aspects on the paper. Eirstly, how they brought entertainment with the learning was interesting. 
Secondly, the use of “Near-Synonyms and Similar-Looking (NSSL)” technology to help students memorizing was 
thoughtful and refreshing. However, the UEEL system could have been developed more efficiently and effectively, if it 
was approached differently. The developer/authors could have used the image instead of the video clip, and adding 
drawing pad for students to draw the image. Drawing the image is the blend of the visual memorization and the Bloom’s 
taxonomy. Evaluation and creating are at the top of the learning process according to Bloom’s taxonomy. By drawing the 
image the evaluation and creating occurs which helps student to memorize vocabulary not only in the short-term memory 
system but also in the long-term memory system. The biggest potential can be found in this program is how students can 
learn without realizing. Students can start the program because of the entertainment but the education occurs without 
students realizing their education. Therefore, the passive students can be also attracted to the education and turn into a 
active learner without knowing. 


KEYWORDS 

Ubiquitous English Vocabulary Learning (UEVL), Near-Synonyms and Similar-Looking (NSSL), Bloom’s taxonomy 


1. INTRODUCTION 

“Ubiquitous English vocabulary learning” (UEVL) is a system developed to help students to experience the 
Systematic Vocabulary Learning process in ubiquitous learning contexts. To explore the perspectives of 
students on the UEVL system, an experiment based on the technology acceptance model was constructed 
(Davis, 1989; Davis, Bagozzi, & Warshaw, 1989). “Near-Synonyms and Similar-Looking” (NSSL) is a 
technology developed in order to find NSSL words and further offer the definitions of the NSSL words being 
considered. They indicated that NSSL words easily lead to confusion, especially when similar -looking words 
have similar translated meanings, such as “transform” and “transfer”. (Huang 1, Huang 2, Huang 3, & Lin, 
2011, p. 4). “Bloom’s taxonomy” is a classification of learning objectives which contains contains (a) 
remembering, (b) understanding, (c) applying, (d) analyzing, (e) evaluating, and (f) creating (Forehand p. 4). 

I found A ubiquitous English vocabulary learning system: evidence of active/passive attitudes vs. 
usefulness/ease-of-use, written by Yueh-Min Huang, Yong-Ming Huang, Shu-hsien Huang, and Yen-Ting 
Lin, very interesting. Although I agreed with many of ideas and suggestions they presented, there were also a 
few methods that I would have approached or developed in a different manner. Authors of this research paper 
impressively approached developing their “Ubiquitous English Vocabulary Learning” (UEVL) system by 
approaching the system in a “partial least square” (Huang 1, Huang 2, Huang 3, & Lin, 2011) way for the 
students’ entertainment while they study. 
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Also, the UEVL worked as one of the most important factors in their learning system, by how it helped 
people to memorize vocabulary definitions easier and in a more enjoyable way than most of the common 
educational systems. However, there were some things that I would have approached differently than they 
did in their paper, such as how they wrote about the process of how their UEVL system works effectively. I 
would have approached this in a more cognitive psychological way. They used “Systematic Vocabulary 
Learning” (SVL) process to explain how their UEVL works effectively (Huang 1, Huang 2, Huang 3, & Lin, 
2011, p. 4). The SVL process is what they created. The idea of SVL is similar to the most popularly proven 
cognitive psychological way of learning process. Bloom’s taxonomy. I would have used Bloom’s taxonomy 
to prove how the UEVL system works in regards to cognitive psychology. The evaluation in terms of lower- 
order and higher-order cognitive skills in Bloom’s taxonomy would have been a more trustworthy way of 
evaluating the UEVL system. It would have been better if they explained the process of learning also in 
regards to neuroscience as well. In other words, they should have explained how visual memorization works 
in the brain. 


2. POSITIVE ASPECTS 

There are also many parts of their paper that I appreciated. I liked how they approached the UEVL system 
(Huang 1, Huang 2, Huang 3, & Lin, 2011, p. 1.) Yueh-Min Huang, Yong-Ming Huang, Shu-hsien Huang, 
and Yen-Ting Lin tried to come up with most intriguing way of students to study vocabulary. Thus, they used 
video clips related to each vocabulary word. The UEVL system had an important role in bringing 
entertainment to studying. Also, they developed “Near-Synonyms and Similar-Looking (NSSL)” technology 
in order to find NSSL words and further offer the definitions of the NSSL words being considered. They 
indicated that NSSL words easily lead to confusion, especially when similar -looking words have similar 
translated meanings, sueh as “transform” and “transfer”. (Huang 1, Huang 2, Huang 3, & Lin, 2011, p. 
4). NSSL technology was brought by the technology from word net (Huang 1, Huang 2, Huang 3, & Lin, 
2011, p. 3). In this way, visual memorization occurs in your brain and helps you to memorize the vocabulary 
easier. NSSL is a good example of combining other’s innovation with their own. 


3. NEGATIVE ASPECTS 

On the other hand, I have a few ideas that I would have approached differently. The authors created the SVL 
process to explain how effective of a process it is in memorization. The SVL process is very similar to the 
cognitive psychology theory Bloom’s taxonomy. On one hand, SVL contains five phases: “(a) encountering, 
(b) getting, (c) understanding, (d) consolidating, and (e) using.” On the other hand. Bloom’s taxonomy 
contains (a) remembering, (b) understanding, (c) applying, (d) analyzing, (e) evaluating, and (f) creating 
(Lorehand p. 4). They are similar but different in some ways. On the other hand, SVL is more like an old 
version of Bloom’s taxonomy. Using a more proven, more popular, and more recent theory would have given 
better information to readers and would be more credible. Also, the way of describing the SVL was neither 
very scientific nor very related to cognitive psychology. Showing how it works in the brain in a cognitive 
way would have given better delivery and trust to the readers. Especially, how the eye movement has a 
relationship with cognition and memory. Although the process of eye movement to the final memorization of 
the word with a video clip in long term visual memory system, it is not written anywhere in the paper. The 
relationship between the posterior parietal cortex, visual cortex, occipital lobes, hippocampus and limbic 
system play an essential role in visual memorization, so a scientific explanation would have been nice. 
(Brady, Konkle, & Alvarez, 2011 p. 23) Moreover, in the program, creating part of Bloom’s taxonomy 
should have been developed more. The part about using their NSSL does not satisfy Bloom’s taxonomy. 
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4. SOLUTION 

The final phase of the process of Bloom’s taxonomy is creating. Which means you have to recall your 
memory and apply it into something. The best way to get better at it is to create a drawing pad and image that 
is related to vocabulary. If you make students to draw the image on the phone while learning the definition, 
because of visual memorization it will not just stay in short term memory but in long term memory system. 
By drawing the image, you focus more than just looking at the video clip or image. Moreover, because an 
image automatically immediately goes to your memory system, if you are drawing an image, when you look 
at the word again, you can recall the image that you drew. Even if you do not remember the definition of the 
vocabulary word, since you remember the image that is related to the word, you can infer from the image and 
figure out the definition. Even if you do not get the exact, correct definition, you can get similar definition 
from the mood of the image. Currently on the market, there are many educational mobile phone applications 
and educational programs that make you look at the image that is related to a vocabulary word, but there few 
to no programs that make you draw the image. By drawing the image, you are basically automatically forced 
to memorize the definition of the vocabulary word. In this way, we can fix the biggest problem of A 
ubiquitous English vocabulary learning system: evidence of active/passive attitudes vs. usefulness/ease-of- 
use. The biggest problem in this paper is that the authors did not address the problem of students who have 
passive attitudes toward education. The paper says that “the passive students were interested in perceived 
ease of use” (Huang 1, Huang 2, Huang 3, & Lin, 2011, p. 2). However, by making them draw an image, 
even the passive students can learn easier with higher chance of memorizing the word. Also, the 
entertainment in the system can bring the passive students into the active students. The student can start out 
the program for an entertainment, but the fact that the vocabulary can be memorized unconsciously and 
automatically by a visual memorization mixed with Bloom’s taxonomy based learning method. Eor this 
reasons, this system has so much potential in it. 


5. CONCLUSION 

A ubiquitous English vocabulary learning system: evidence of active/passive attitudes vs. usefulness/ease -of- 
use is research that possessed a great concept of putting entertainment and making it easier to memorize 
vocabulary. It contained a lot of clever devices to help study words and make language learning easier 
through means such as NSSR technology and usage of SVL process. However, some downfalls include not 
utilizing Bloom’s taxonomy and not explaining the cognitive psychological way of how visual memorization 
works. Most importantly, the biggest flaw was how “passive students were interested in perceived ease of use” 
(Huang 1, Huang 2, Huang 3, & Lin, 2011, p. 2) therefore leaving passive students at an educational loss as 
they are not learning. Thus, if the approach to visual memorization in the program changes, images rather 
than video clips and drawing the image, the problem of the passive students’ attitude could be changed 
because of the entertainment and education occurring students even realizing it. The ubiquitous educational 
programming has huge potential. Therefore, if the program can be developed after fixing the flaw, the 
potential could be limitless. 
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